ML job failures are silently dropped (no retry, not counted as failed) #32815

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

ML job failures are silently dropped (no retry, not counted as failed); server times out ML requests at 300 s while ROCm worker allows 900 s

If a machine-learning request fails (ML container restarting, slow, or OOM-killed), the job is lost without a trace:

1. No retry. config.repository.js sets BullMQ defaultJobOptions: { attempts: 1 }.
2. Not counted as failed. JobService.onJobRun catches the handler error and only emits JobError, so BullMQ marks the job completed. The Jobs page shows failed: 0 and an empty queue, but the assets stay unprocessed (asset_job_status."ocrAt" is still NULL). Nothing tells you to run "Missing" again.
3. Timeout mismatch on ROCm. MachineLearningRepository.predict() uses the global fetch() with no dispatcher, so undici's default headersTimeout of 300 s applies. The ML service raises its gunicorn worker timeout to 900 s on ROCm (immich_ml/config.py: default_worker_timeout() → 900 if DEVICE == "rocm" else 300). The server gives up on requests the worker is still allowed to finish, and the job is then dropped as in 1 and 2.

• I restarted the ML container while OCR jobs were queued. About 370 jobs "completed" within 2 s, and each one logged failed for all URLs. The Jobs page showed failed: 0.
• An overnight OCR backlog on ROCm hit UND_ERR_HEADERS_TIMEOUT 115 times in 24 h. The queue drained to 0, but 444 images had no OCR. A second "Missing" run picked them up.

Expected: ML failures are retried with backoff, or the job is held while ML is unhealthy. At a minimum, a job that really fails should go to failed. The server's ML request timeout matches the worker timeout (an undici Agent with headersTimeout/bodyTimeout ≥ the worker timeout) or can be configured.

The OS that Immich Server is running on

Unraid 7.3.4 (Linux 6.18.58), Docker

Version of Immich Server

v3.3.5

Version of Immich Mobile App

n/a

Platform with the issue

• Server
• Web
• Mobile

Device make and model

No response

Your docker-compose.yml content

# Stock release docker-compose.yml (unchanged), plus this docker-compose.override.yml:
services:
  database:
    command: ["postgres", "-c", "config_file=/etc/postgresql/postgresql.conf", "-c", "shared_preload_libraries=vchord.so"]
  immich-machine-learning:
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-rocm
    devices:
      - /dev/kfd
      - /dev/dri
    mem_limit: 12g        # was 10g during the incidents
  immich-server:
    devices:
      - /dev/dri:/dev/dri # VAAPI on the RX 9070 (renderD129)
    volumes:
      - /mnt/cache/appdata/immich-thumbs:/data/thumbs

Your .env content

UPLOAD_LOCATION=/mnt/user/Docs/Pics/immich
DB_DATA_LOCATION=/mnt/cache/appdata/postgresql/data
IMMICH_VERSION=release
DB_PASSWORD=xyz
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

Silent drop (any hardware):
1. Have assets without OCR (or a small library you can re-run OCR "All" on).
2. docker stop immich_machine_learning
3. Administration → Jobs → OCR → Missing
4. The queue drains within seconds, counters show failed: 0, and the server logs
   "Unable to run job handler (Ocr): ... failed for all URLs" once per asset.
5. select count(*) from asset_job_status where "ocrAt" is null;   -- unchanged
   (Steps 2–5 are taken from the restart incident above plus the code path; I have not run them as a
   separate test.)

300 s client timeout (run inside the immich-server image, Node v24.22.1):
1. Save as t.mjs and run `node t.mjs`:
   import http from "node:http";
   http.createServer((q, s) => setTimeout(() => s.end("late"), 330000)).listen(39999);
   const t0 = Date.now();
   try { await fetch("http://127.0.0.1:39999/predict", { method: "POST", body: "x" }); }
   catch (e) { console.log((Date.now() - t0) / 1000, e.cause?.code); }
2. Output: 300.748 UND_ERR_HEADERS_TIMEOUT

Relevant log output

``shell
[Nest] 7  - 10/28/2026, 8:06:37 PM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://immich-machine-learning:3003" failed
[Nest] 7  - 10/28/2026, 8:06:37 PM    WARN [Microservices:MachineLearningRepository] [TypeError: fetch failed] {
  [cause]: HeadersTimeoutError: Headers Timeout Error
      at FastTimer.onParserTimeout [as _onTimeout] (node:internal/deps/undici/undici:7763:32)
    code: 'UND_ERR_HEADERS_TIMEOUT'
}
[Nest] 7  - 10/28/2026, 8:06:37 PM     LOG [Microservices:MachineLearningRepository] Machine learning server became unhealthy (http://immich-machine-learning:3003).
[Nest] 7  - 10/28/2026, 8:06:37 PM   ERROR [Microservices:{"id":"7ce52bc7-8475-468f-a962-80c487007ce5"}] Unable to run job handler (Ocr): Error: Machine learning request '{"ocr":{...}}' failed for all URLs
    at MachineLearningRepository.predict (file:///usr/src/app/server/dist/repositories/machine-learning.repository.js:114:15)
    at async MachineLearningRepository.ocr (file:///usr/src/app/server/dist/repositories/machine-learning.repository.js:147:26)
    at async OcrService.handleOcr (file:///usr/src/app/server/dist/services/ocr.service.js:42:28)
    at async JobService.onJobRun (file:///usr/src/app/server/dist/services/job.service.js:75:30)

Additional information

The slow ROCm requests that hit the timeout come from issue #32816 (ML worker memory growth → swap).
Possibly related: discussion #32447.

(Disclaimer: Report written with Claude's help to identify and reproduce exact failure I experienced)