ROCm ML: every compiled OCR input shape stays resident in host RAM and doesn't release under load #32816

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

On the ROCm image, OrtSession.for_shape() (immich_ml/sessions/ort.py) builds one MIGraphX graph per input shape and caches it in self.graphs with no eviction. ModelCache only evicts a whole model after model_ttl (300 s) of idle time, and that never happens during a backlog. PP-OCRv5 uses 7 detection shapes and 12 recognition shapes (OCR_RECOGNITION_WIDTHS 224–2048 × batch 1 and 6). Each shape loaded costs hundreds of MB to over 1 GB of host RAM, so the worker grows to about 6–10 GB and stays there.

In my case with mem_limit at 10 GB (later 12 GB), a real OCR backlog pushed the worker into swap within about 10 minutes. Throughput fell from ~3,000 to ~10 per 5 min, and requests went over the server's 300 s timeout (issue #32815). Each container restart brought back full speed for about 4–5k assets.

Expected: per-shape graphs are capped or LRU-evicted, or idle shapes are evicted, not only whole models. If not, the host-RAM footprint of ROCm OCR should be considered.

The OS that Immich Server is running on

Unraid 7.3.4 (Linux 6.18.58), Docker

Version of Immich Server

v3.3.5

Version of Immich Mobile App

n/a

Platform with the issue

• Server
• Web
• Mobile

Device make and model

No response

Your docker-compose.yml content

# Stock release docker-compose.yml (unchanged), plus this docker-compose.override.yml:
services:
  database:
    command: ["postgres", "-c", "config_file=/etc/postgresql/postgresql.conf", "-c", "shared_preload_libraries=vchord.so"]
  immich-machine-learning:
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-rocm
    devices:
      - /dev/kfd
      - /dev/dri
    mem_limit: 12g        # was 10g during the incidents
  immich-server:
    devices:
      - /dev/dri:/dev/dri # VAAPI on the RX 9070 (renderD129)
    volumes:
      - /mnt/cache/appdata/immich-thumbs:/data/thumbs

Your .env content

UPLOAD_LOCATION=/mnt/user/Docs/Pics/immich
DB_DATA_LOCATION=/mnt/cache/appdata/postgresql/data
IMMICH_VERSION=release
DB_PASSWORD=xyz
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. ROCm ML image, OCR enabled with defaults (MACHINE_LEARNING_MAX_BATCH_SIZE__OCR = 6).
2. Find the gunicorn worker PID and record RssAnon / RssShmem from /proc/<pid>/status.
3. Send OCR POST /predict requests with images that produce different numbers and lengths of text lines
   (used synthetic text images: 1×8, 1×40, 6×8, 6×40 and 8×70 lines×chars), or run OCR "All" on a varied library.
4. RssAnon grows by hundreds of MB for each new shape and never comes down while requests keep arriving.

Measured on the live worker:
| Request                              | New graph | 1st request | RssAnon | RssShmem |
| baseline (OCR loaded, 1 det shape)   |           |             | 2.59 GB | 0.43 GB  |
| 1 line × 8 chars                     | yes       | 1.4 s       | 2.82 GB | 0.48 GB  |
| 1 line × 40 chars                    | yes       | 2.9 s       | 3.18 GB | 0.54 GB  |
| 6 lines × 8 chars                    | yes       | 3.5 s       | 3.65 GB | 0.60 GB  |
| 6 lines × 40 chars                   | yes       | 8.1 s       | 4.83 GB | 0.66 GB  |
| 8 lines × 70 chars                   | yes       | 9.8 s       | 6.20 GB | 0.74 GB  |
| repeat 6×40, repeat 1×8              | no        | 0.1 s       | 6.20 GB | 0.78 GB  |

Relevant log output

# watcher on the worker during the backlog (10 GB limit; done_15m = OCR jobs finished in 15 min)
2026-10-29 02:58:47 ocr waiting=482 done_15m=3708 worker[RssAnon:2.5G RssShmem:9.4G VmSwap:3.8G]
2026-10-29 03:03:47 ocr waiting=475 done_15m=574  worker[RssAnon:2.4G RssShmem:9.5G VmSwap:3.9G]
2026-10-29 03:08:48 ocr waiting=473 done_15m=121  worker[RssAnon:2.4G RssShmem:9.5G VmSwap:3.9G]
# after docker restart immich_machine_learning: the rest of the queue finished within 5 min

Additional information

• New detection shapes behave the same way (+0.34 to +0.53 GB each).
• Thirty identical requests on a shape that is already loaded add nothing.
• After the 300 s idle unload, the worker drops back to about 0.25 GB.
• So this is graphs being kept resident, not a per-request leak.
• It is not GPU spill: GTT in use was about 50 MB throughout. VRAM is shared with other workloads.
• ROCm 7.2.0.
• MACHINE_LEARNING_MAX_BATCH_SIZE__OCR=1 should avoid the six batch-6 recognition graphs, the largest ones (batch6_width2048 is 772 MB on disk). Not tested yet.

(Disclaimer: Report written with Claude's help to identify and reproduce exact failure I was having)