[willtrking]

Summary

With IMMICH_STORAGE_BACKEND=s3, the storage template did nothing. S3 uploads stayed at upload/<userId>/… forever, the Migration job skipped S3 thumbnails and encoded videos, and the S3 docs said the opposite. This PR makes the template work in the bucket.

S3 has no rename (RenameObject exists only for S3 Express One Zone directory buckets), so a move is:

1. a server-side CopyObject (UploadPartCopy above 5 GiB), so no bytes pass through Gallery;
2. a size check;
3. the database pointed at the new key;
4. a delete of the old key.

move_history gives the same crash recovery upstream's disk move has. A file always stays in the backend that holds it.

Spec: specs/2026-10-20-storage-template-s3-moves-design.md records the design, the decisions and their reasons, and the known limits. It also marks the parts of specs/2026-03-02, 2026-04-19 and 2026-04-22 that it supersedes.

Behaviour

• Uploads move to their template path (library/<storage label or user id>/…) after metadata extraction, as on disk.
• Upgrade exception: an install that upgrades with photos or videos already in S3 and the template enabled waits.
  • Its S3 originals and sidecars keep their upload keys until an admin runs Storage Template Migration once. That run moves everything, and from then on uploads move automatically.
  • Uploads keep working normally in the meantime, and a warning is logged on every start while waiting.
  • The decision is made once, on the first start, under a database lock, and stored in system_metadata as storage-template-s3-state.
  • Fresh installs, installs whose template was off at upgrade, and disk files are not affected.
• A manual Storage Template Migration on an S3 install also queues upstream's Migration job (FileMigrationQueueAll), unless it is already running. That job re-paths thumbnails, previews, full-size images, encoded videos and person thumbnails, in S3 and on disk. The Migration job now handles S3 files too.
• Verification matches upstream's disk move: a size check after each copy, and SHA-1 (hashVerificationEnabled) only when finishing an interrupted move.
• Safety:
  • A destination counts as taken when either backend holds it.
  • A sidecar waits while its original's move is unfinished, and never overwrites an existing sidecar.
  • The template root ends in /, so EXIF-driven templates cannot escape into a sibling library folder.
  • On a size mismatch, only an original's copy is deleted.
  • The mover is S3-only: without S3 configured it warns and leaves the file.
• Backends:
  • Copies keep the source's standard S3 storage class.
  • Gateways without UploadPartCopy fall back to CopyObject.
  • Disk copies keep atime/mtime.
  • Disk writes go to a temp file that is renamed into place.
• Accounting and cleanup:
  • Storage usage counts the S3 library/<label>/ prefix.
  • User deletion removes that prefix, but skips it when another user's files are stored there.
  • The storage migration compares sizes before reusing an existing target.
• Android motion videos: embedded motion videos in S3 (encoded-video/…-MP.mp4) are not templated, as on disk. This also fixes includeEmbeddedVideos for S3 downloads.

Layering

Fork logic lives in server/src/gallery/:

• utils/s3-file-move.ts
• utils/s3-key.ts, with toS3Key and isInStorageFolder
• utils/storage-backends.ts, the backend registry: it imports no service, so the new code needs no lazy imports
• utils/storage-template-s3.ts
• services/storage-template-s3.service.ts

Upstream files, diffed against upstream v3.3.5 (upstream.version in branding/config.json), on main and on this branch:

Upstream file	main	This PR	Why
server/src/services/storage-template.service.ts	+10 −1	+17 −4	S3 skip removed; calls into the fork module for the library root, the destination check and the upgrade gate
server/src/cores/storage.core.ts	+52 −5	+64 −6	the early return for keys becomes one call into the S3 mover; isAndroidMotionPath is one isInStorageFolder call
server/src/services/storage.service.ts	+49 −2	+29 −2	shrinks: the backend instances moved to a leaf registry; four one-line delegates remain
server/src/services/user.service.ts	+58 −9	+51 −9	shrinks: the fork-owned S3 cleanup block reads its prefixes from storage-usage.ts
server/src/enum.ts	+210	+211	one SystemMetadataKey line
server/src/types.ts	+210 −6	+211 −6	the state's type, one line
docs/docs/administration/storage-template.mdx	none	+8	an S3 note: the template applies in the bucket, the upgrade exception, and links to the S3 storage page

The upstream spec files grow with tests of the new behaviour: storage-template.service.spec.ts (+1013 −1), storage.core.spec.ts (+132 −1), person.service.spec.ts (+91 −31), media.service.spec.ts (+65), user.service.spec.ts (+50) and storage.service.spec.ts (+1). Four more stay the same size, because they only swap StorageService's backend statics for the registry: the asset-media, auth and metadata service specs, and the medium asset-media spec.

storage-template-s3-state follows the fork's existing system_metadata state keys, and scripts/revert-to-immich.sql deletes it. move_history keeps Immich's meaning, and its rows are transient. Every changed file is covered by docs/fork/ownership.yml (the preflight coverage check passes).

No schema, migration, endpoint, OpenAPI or i18n changes. An earlier revision added a JobName value, which upstream's check-openapi rejects as a breaking change; the derivative pass reuses upstream's Migration job instead.

Testing

• On this branch, rebased onto main (Immich v3.3.5): the server type-check passes, and the full server unit suite passes on Node 24 (7,002 tests); ESLint and Prettier are clean on the changed files. The suite needs NODE_OPTIONS=--max-old-space-size=6144: at the default ~2 GB heap, shared-space.service.spec.ts runs out of memory on main too. commit-autolink-check and the ownership coverage check pass.
• CI on the same commits before the rebase (fork PR willtrking/gallery 1, Immich v3.3.3 base): every check passed except the changelog-label check. That covered:
  • server unit tests and lint;
  • medium tests, among them the new hasS3Originals() spec;
  • SQL schema checks;
  • the web and Server & CLI E2E suites;
  • the revert-to-Immich validation.
• Not run yet: the Storage Migration Tests workflow, which runs only on schedule or dispatch, never on pull requests. It covers:
  • the MinIO integration spec for the new backend copy/stat (keys that need encoding, overwriting, missing objects, part-by-part copy, and the fallback for gateways without UploadPartCopy);
  • the template-s3-* E2E phases, including template-s3-upgrade-prepare → server restart → template-s3-upgrade-pending, which exercises the real upgrade decision.

Known limits

All of these are recorded in the design doc.

• A clearing manual run whose S3 moves all fail still ends the wait. Each failure is logged.
• hasS3Originals() scans the asset table once, at first start, on a large install without S3 originals.
• Crash windows can leave an orphaned object. Integrity checks still skip S3 (Extend integrity checks to S3-backed storage #685).
• RC images built from this branch's first commits already moved S3 uploads automatically. They decide "waiting" on their first start after updating, so testers should run the job once.

---

Generated by Claude Code