Skip to main content

D86 — Documents vault direct-to-object-storage transfer: app-signed issue/confirm envelope around presigned S3 URLs (RESOLVED, extends D43)

Architecture decision record, relocated verbatim from the retired single-file docs/tenancy/ decision log. Status, thematic clusters, and how to record a new ADR: the decision log index.

The vault's proxied upload (D43) pushes every byte through the app (POST /api/v1/documents → encrypt → local/private disk), which caps practical file size at the PHP/proxy request envelope and makes the app a bandwidth middleman. Decision: an opt-in direct-transfer flow where the app issues short-lived presigned URLs and owns the metadata/lifecycle, and the BYTES move directly between the client and an S3-compatible bucket — config-gated OFF by default (documents.direct_transfer.enabled; both endpoints refuse 409 DIRECT_TRANSFER_DISABLED, and the flag is a kill switch for the whole surface including the confirm). Shape:

  • Initiate (POST /documents/direct-uploads → InitiateDirectDocumentUploadCommand, the same documents.upload permission as the proxied store): creates the metadata row via the bus in the tenant's RLS scope — status awaiting_upload (a new pre-byte DocumentStatus), a server-derived object key (direct/<uuid>.bin; never client-chosen, never exposed — the URL embeds it), storage_encryption = server_side (D87), no mime/sha256 yet (both nullable until the bytes land) — and returns a presigned PUT (documents.direct_transfer.upload_url_ttl, 300s) minted through the DirectTransferUrlSigner seam (tests bind a fake; the real signer wraps the S3 driver's temporaryUploadUrl, proven against MinIO per D82).
  • Confirm (POST /documents/{uuid}/direct-uploads/complete → CompleteDirectDocumentUploadCommand, idempotent): verifies the object LANDED at the upload key (missing ⇒ 409 DIRECT_UPLOAD_INCOMPLETE, row stays awaiting), then server-side COPIES it to a distinct CONFIRMED key (direct/<key>/confirmed.bin — a key NO presigned PUT is ever issued for) and streams THE CONFIRMED COPY once server-side (8 KB chunks) to (a) count the ACTUAL size against documents.max_bytes (aborting early past the cap), (b) sniff the MIME from the leading bytes (the D43 trust-the-bytes rule), and (c) compute the sha256 integrity anchor every read path + the integrity sweep re-verify. The copy is the swap-window (TOCTOU) control: the presigned PUT stays technically writable for its full TTL even after confirm, so verifying and serving the upload key itself would let a client swap the object post-confirm/post-scan and have the swapped bytes served; instead the confirmed copy is what is verified, persisted as the document's path, and read by every downstream path (serve/forward/scan/sweep) — a post-confirm PUT to the old upload key mutates an inert object. On admit the now-inert upload key is deleted (best-effort, checked + logged). A violation deletes BOTH objects (checked + logged) and refuses 422 DOCUMENT_REJECTED (the vault never admits the bytes). On admit: stored, then the normal downstream (D88 quarantine/scan when enabled, else the D43 forward) — dispatched afterCommit(), preserving store-before-forward. Transaction shape: the command is WithoutTransaction (the LogoutClientCommand opt-out precedent) — streaming a 25 MB object must not run inside a DB transaction holding a row lock, so the handler does all object-storage I/O lock-free and then opens its OWN short lockForUpdate transaction to re-assert awaiting_upload (the race arbiter — the loser returns the admitted document idempotently) and flip the state. NOTE the app deliberately reads the bytes ONCE at confirm (bucket→app, not client→app): giving up the sniff + hash would gut the vault's integrity model, and the flow's point is to offload the CLIENT transfer path, not to blind the vault. The synchronous ContentPreScanner tripwire does NOT run on this path (inert for the binary allowed-mime set anyway); the D88 malware scan is the direct path's content control.
  • Download / access logging (the load-bearing part): IssueDocumentDownloadUrlQuery still mints the ~90s app-signed stream route — the signature stays the credential and the route stays the single choke point. For a server_side document the streaming controller logs the serve to document_accesses FIRST, then 302-redirects to a short-lived presigned GET (download_url_ttl) minted through the same signer seam. A raw presigned GET handed straight to clients would bypass the per-serve access log (D6) and the DLP egress capture — the redirect preserves both (the EgressesData volume reads the stored size for a redirect serve, so egress accounting stays truthful). The presigned GET signs the ORIGINAL filename + sniffed MIME into the S3 response-content-disposition/response-content-type overrides (RFC 6266, the same escaping as the envelope path's streamDownload), so a redirect download presents like a streamed one — never the raw direct/…confirmed.bin key with the client-set upload content type. A missing object at serve time fails closed as DOCUMENT_INTEGRITY_FAILURE and is logged integrity_failed, exactly like a missing envelope blob. The serve does NOT re-hash the whole object per request (that would re-proxy the bytes); the confirm-time sha256 (of the confirmed copy — the only key ever served) is re-verified by the daily integrity sweep and any server-side read.
  • Size enforcement point: at presign-issue time against the client-DECLARED size (fast feedback), and authoritatively at CONFIRM time against the actual object — a presigned PUT cannot carry a content-length-range condition (that is presigned-POST policy machinery, and Laravel's temporaryUploadUrl presigns a PutObject with no signed Content-Length). Production should pair this with a bucket lifecycle rule cleaning direct/ objects older than ~1 day (abandoned PUTs) — deploy config, documented in the tracking doc; upload_expires_at is the in-app handle for a future abandoned-row prune.

Deliberately NOT built: a webhook/S3-event-driven auto-confirm (the explicit client confirm keeps the flow provider-agnostic and testable), any client choice of key/bucket, and a public-read bucket posture (the bucket stays private; every GET is presigned + app-logged).

← Decision log index