Skip to main content

D86 — Documents vault direct-to-object-storage transfer: app-signed issue/confirm envelope around presigned S3 URLs (RESOLVED, extends D43)

Architecture decision record, relocated verbatim from the retired single-file docs/tenancy/ decision log. Status, thematic clusters, and how to record a new ADR: the decision log index.

The vault's proxied upload (D43) pushes every byte through the app (POST /api/v1/documents → encrypt → local/private disk), which caps practical file size at the PHP/proxy request envelope and makes the app a bandwidth middleman. Decision: an opt-in direct-transfer flow where the app issues short-lived presigned URLs and owns the metadata/lifecycle, and the BYTES move directly between the client and an S3-compatible bucket — config-gated OFF by default (documents.direct_transfer.enabled; both endpoints refuse 409 DIRECT_TRANSFER_DISABLED, and the flag is a kill switch for the whole surface including the confirm). Shape:

  • Initiate (POST /documents/direct-uploads → InitiateDirectDocumentUploadCommand, the same documents.upload permission as the proxied store): creates the metadata row via the bus in the tenant's RLS scope — status awaiting_upload (a new pre-byte DocumentStatus), a server-derived object key (direct/<uuid>.bin; never client-chosen, never exposed — the URL embeds it), storage_encryption = server_side (D87), no mime/sha256 yet (both nullable until the bytes land) — and returns a presigned PUT (documents.direct_transfer.upload_url_ttl, 300s) minted through the DirectTransferUrlSigner seam (tests bind a fake; the real signer wraps the S3 driver's temporaryUploadUrl, proven against MinIO per D82).
  • Confirm (POST /documents/{uuid}/direct-uploads/complete → CompleteDirectDocumentUploadCommand, idempotent): verifies the object LANDED at the upload key (missing ⇒ 409 DIRECT_UPLOAD_INCOMPLETE, row stays awaiting), then server-side COPIES it to a distinct CONFIRMED key (direct/<key>/confirmed.bin — a key NO presigned PUT is ever issued for) and streams THE CONFIRMED COPY once server-side (8 KB chunks) to (a) count the ACTUAL size against documents.max_bytes (aborting early past the cap), (b) sniff the MIME from the leading bytes (the D43 trust-the-bytes rule), and (c) compute the sha256 integrity anchor every read path + the integrity sweep re-verify. The copy is the swap-window (TOCTOU) control: the presigned PUT stays technically writable for its full TTL even after confirm, so verifying and serving the upload key itself would let a client swap the object post-confirm/post-scan and have the swapped bytes served; instead the confirmed copy is what is verified, persisted as the document's path, and read by every downstream path (serve/forward/scan/sweep) — a post-confirm PUT to the old upload key mutates an inert object. On admit the now-inert upload key is deleted (best-effort, checked + logged). A violation deletes BOTH objects (checked + logged) and refuses 422 DOCUMENT_REJECTED (the vault never admits the bytes). On admit: stored, then the normal downstream (D88 quarantine/scan when enabled, else the D43 forward) — dispatched afterCommit(), preserving store-before-forward. Transaction shape: the command is WithoutTransaction (the LogoutClientCommand opt-out precedent) — streaming a 25 MB object must not run inside a DB transaction holding a row lock, so the handler does all object-storage I/O lock-free and then opens its OWN short lockForUpdate transaction to re-assert awaiting_upload (the race arbiter — the loser returns the admitted document idempotently) and flip the state. NOTE the app deliberately reads the bytes ONCE at confirm (bucket→app, not client→app): giving up the sniff + hash would gut the vault's integrity model, and the flow's point is to offload the CLIENT transfer path, not to blind the vault. The synchronous ContentPreScanner tripwire does NOT run on this path (inert for the binary allowed-mime set anyway); the D88 malware scan is the direct path's content control.
  • Download / access logging (the load-bearing part): IssueDocumentDownloadUrlQuery still mints the ~90s app-signed stream route — the signature stays the credential and the route stays the single choke point. For a server_side document the streaming controller logs the serve to document_accesses FIRST, then 302-redirects to a short-lived presigned GET (download_url_ttl) minted through the same signer seam. A raw presigned GET handed straight to clients would bypass the per-serve access log (D6) and the DLP egress capture — the redirect preserves both (the EgressesData volume reads the stored size for a redirect serve, so egress accounting stays truthful). The presigned GET signs the ORIGINAL filename + sniffed MIME into the S3 response-content-disposition/response-content-type overrides (RFC 6266, the same escaping as the envelope path's streamDownload), so a redirect download presents like a streamed one — never the raw direct/…confirmed.bin key with the client-set upload content type. A missing object at serve time fails closed as DOCUMENT_INTEGRITY_FAILURE and is logged integrity_failed, exactly like a missing envelope blob. The serve does NOT re-hash the whole object per request (that would re-proxy the bytes); the confirm-time sha256 (of the confirmed copy — the only key ever served) is re-verified by the daily integrity sweep and any server-side read.
  • Size enforcement point: at presign-issue time against the client-DECLARED size (fast feedback), and authoritatively at CONFIRM time against the actual object — a presigned PUT cannot carry a content-length-range condition (that is presigned-POST policy machinery, and Laravel's temporaryUploadUrl presigns a PutObject with no signed Content-Length). Production should pair this with a bucket lifecycle rule cleaning direct/ objects older than ~1 day (abandoned PUTs) — deploy config, documented in the tracking doc; upload_expires_at is the in-app handle for a future abandoned-row prune.

Deliberately NOT built: a webhook/S3-event-driven auto-confirm (the explicit client confirm keeps the flow provider-agnostic and testable), any client choice of key/bucket, and a public-read bucket posture (the bucket stays private; every GET is presigned + app-logged).

D86.1 — Enablement + the local browser-reachable presign seam (2026-08-04)​

Enablement: documents.direct_transfer.enabled (and the presign-disk key below) are now env-driven per environment through the ROOT config/documents.php (php artisan vendor:publish --tag=documents-config) rather than left as a literal-only module default — local dev and the AWS envs each set DOCUMENTS_DIRECT_TRANSFER_ENABLED independently (see app-modules/documents/README.md § Enablement).

Problem found enabling it locally: presigned URLs are minted against direct_transfer.disk's configured endpoint — locally s3-documents, whose AWS_ENDPOINT is the container-internal http://minio:9000 (.env.example). A HOST browser (the actual client of a direct upload) cannot resolve that hostname. SigV4 signs the Host header into the URL, so rewriting a minted URL after signing (e.g. Laravel's temporary_url config hook) invalidates the signature — the fix has to happen at signing time, not after.

Decision: add documents.direct_transfer.presign_disk (nullable, default null) — a disk to presign against IN PLACE OF disk, for URL GENERATION ONLY. Every real object-storage call (exists/copy/read/delete, in InitiateDirectDocumentUploadHandler + CompleteDirectDocumentUploadHandler) is unaffected and always uses direct_transfer.disk directly. This is safe because presigning is an offline SigV4 computation over the disk's key/secret/region/bucket — never a network call — so a disk config identical to s3-documents except its endpoint produces a URL valid from wherever that endpoint is reachable, against the exact same bucket/objects. FilesystemDirectTransferUrlSigner resolves the substitution once, in a private presignAdapter() shared by signUpload + signDownload.

Locally: config/filesystems.php ships a mirrored s3-documents-presign disk (same key/secret/region/bucket as s3-documents; endpoint = DOCUMENTS_VAULT_PRESIGN_ENDPOINT, default http://localhost:19000 — MinIO's published host-port S3 API, docker-compose.yml), and .env.example/.env set DOCUMENTS_VAULT_PRESIGN_DISK=s3-documents-presign. A client PUT/GET still lands on the SAME MinIO container either way (the published port forwards into it) — only the signed Host in the URL differs. presign_disk stays unset in the AWS envs: the app and the browser already reach the same S3 endpoint there, so no substitution is needed and nothing about the D86/D87 posture changes.

D86.1 addendum — the upload-intent endpoint: transport is a backend decision (2026-08-04)​

D86 shipped direct-transfer as an opt-in flow the client had to know about: initiate direct-uploads directly, and on the OFF-by-default kill switch, fall back to the proxied POST /documents upload — a 409 DIRECT_TRANSFER_DISABLED dance the FE had to implement itself. Product decision: the transport is the backend's call, not the frontend's. The FE now calls a single entry point, POST /documents/upload-intents (DocumentUploadIntentController), and follows whatever instruction comes back — it never branches on the kill switch itself. The request is { original_name, size_bytes, declared_mime, kind? }: original_name/size_bytes/kind deliberately reuse the sibling direct-uploads initiate endpoint's own field names + rules verbatim — two sibling endpoints on the same resource must not name the same concept differently. kind (optional) threads through to InitiateDirectDocumentUploadCommand on the direct branch exactly like the sibling endpoint — it feeds DocumentPurposeMap, the provider purpose hint on forward, so dropping it silently degrades KYB/KYC uploads — but needs no threading on proxied, whose own follow-up POST /documents call carries kind itself.

Routing rule: a new pure policy query, ResolveDocumentUploadTransportQuery → ResolveDocumentUploadTransportHandler (reads only documents.direct_transfer.enabled, touches no Eloquent/storage), resolves always direct when the flag is true, proxied otherwise — no size threshold. Simpler than a size-tiered rule: the flag already exists to gate the whole surface (a deploy without a provisioned bucket must never receive a direct instruction), and once it's on there is no cost reason to keep small uploads proxied.

  • direct — the controller dispatches the EXISTING, unchanged InitiateDirectDocumentUploadCommand (D86, no behavior change) and shapes { mode: "direct", document, upload: { url, headers, expires_at }, complete_url } — complete_url is the named direct-uploads.complete route, so the FE never constructs it.
  • proxied — no document row is created (unlike the direct branch, which always creates the awaiting_upload row at intent time). The response just names the endpoint: { mode: "proxied", upload_endpoint }; the client still POSTs the actual bytes there.
  • Two fast-fail gates move earlier and apply to both branches — at the INTENT step. A declared size_bytes over documents.max_bytes now 422s DOCUMENT_REJECTED at the intent step regardless of which branch would have been picked — the direct branch's own initiate handler already fast-failed this (D86), but the proxied branch previously had no equivalent pre-flight check; the intent controller now enforces the same cap directly before asking the transport query, for symmetry. That symmetry holds only at the intent step: a proxied resolution's SEPARATE follow-up POST /documents call re-validates size via Laravel's file max: rule, which 422s VALIDATION_ERROR (with a source.pointer) — not DOCUMENT_REJECTED. The FE should treat either code as the too-large signal. A declared_mime outside documents.allowed_mimes is rejected the same way at the intent step, but is ADVISORY ONLY — D43's trust-the-bytes rule is unchanged, so the sniffed bytes at store/confirm time remain the sole AUTHORITATIVE MIME check; a client that declares a compliant type but uploads something else is still caught later, exactly as before this endpoint existed. Confirm-time verification (the actual object, sniffed MIME, sha256) remains the sole AUTHORITATIVE check on the direct path for both size and MIME — the intent-time gates are only fast client-declared checks, same posture as D86's own presign-issue check.
  • Observability asymmetry. A proxied intent leaves NO persisted trace (nothing is created until the follow-up upload lands). A direct intent ALWAYS leaves an awaiting_upload row the moment it resolves, even if the client abandons the flow before ever PUTting bytes (same pre-existing behavior as calling direct-uploads.store directly — see "Abandoned direct-upload rows" in the README's Deferred section). Support triage: "no row" after a reported failure is normal for an abandoned proxied attempt, but for a direct attempt it usually means the row IS there, still awaiting_upload.

Positioning, not removal. The direct-uploads.store/.complete pair from D86 REMAINS directly callable — nothing about their contract changed, and the intent endpoint's own complete_url routes straight through .complete. They are now documented as legacy/internal: a new integration calls upload-intents.store only.

No 409 DIRECT_TRANSFER_DISABLED is reachable from the intent action itself — the transport query and the initiate handler it may dispatch both read the same static documents.direct_transfer.enabled config value within a single request (no Octane, no mid-request config mutation), so they cannot disagree in one request-response cycle; the "no fallback dance required" claim holds there. The race is cross-request: the flag CAN flip between an intent call that promised direct and the client's LATER PUT/complete — that pre-existing 409 still surfaces from direct-uploads.complete (or a retried .store), unchanged from D86. A client that hits it should discard the stale instruction and request a fresh intent rather than retry the same one.

← Decision log index