Integrity report jobs fail with CONNECTION_DESTROYED when they start on the checksum job's stop tick (break out of a Kysely stream) #32712

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

With a library large enough that the checksum integrity job always hits its timeLimit, the two integrity report jobs that are queued behind it (IntegrityUntrackedFilesQueueAll and IntegrityMissingFilesQueueAll) can fail with write CONNECTION_DESTROYED <db>:5432 at the exact second the checksum job logs Reached stop criteria.

Setup where it happens:

• about 140,000 assets, checksum job averages 120 to 130 ms per asset, so one hour covers roughly 28,000 and the job stops on timeLimit every night
• the three integrity crons left on their defaults (0 3 * * *), so all three jobs are queued at 03:00 on the integrityCheck queue and the two report jobs start the moment the checksum job finishes, at 04:00:00

What I saw (Immich v3.1.0, late August):

• 28 Aug: Reached stop criteria at 04:00:00, both report jobs failed at 04:00:02 with write CONNECTION_DESTROYED database:5432
• 29 Aug: identical sequence, both report jobs succeeded
• 30 Aug: Reached stop criteria at 04:00:00, both report jobs failed at 04:00:00
• the failing queries report durationMs between 0.19 and 1.44 ms, so no round trip to Postgres ever happened. postgres.js raises CONNECTION_DESTROYED client side only, from its own terminate() path or when a query is handed to a connection that has already been terminated (src/connection.js in postgres 3.4.9)
• Postgres itself logged nothing at 04:00 on either night (no disconnect, no error, no restart), max_connections 100 with 3 in use, no timeouts configured
• the 02:00 in-app database dump and the host backups all ran well before 04:00 and do not touch the containers
• on earlier nights when the report jobs happened to run at 03:00 to 03:43 (before the checksum job reached its stop tick) they always succeeded
• the net effect was that integrity_report had not refreshed since 24 Jul, because the report jobs were dying before doing anything

Workaround that has held since 30 Aug (zero failures through 24 Oct, now on v3.3.1): move integrityChecks.untrackedFiles.cronExpression and integrityChecks.missingFiles.cronExpression to 0 2 * * *, leaving the checksum job at 03:00. The checksum job still logs Reached stop criteria at 04:00:00 every night, but nothing else is starting a query at that instant any more.

Where I think it comes from (not confirmed by debugging the driver, so treat this as a pointer):

• handleChecksumFiles in server/src/services/integrity.service.ts iterates this.integrityRepository.streamAssetChecksums(startMarker), which is a Kysely .stream() (server/src/repositories/integrity.repository.ts, streamAssetChecksums), and when the stop criteria are met it breaks out of the for await (

  immich/server/src/services/integrity.service.ts

  Lines 574 to 587 in 256eb55

  	for await (const { originalPath, checksum, createdAt, assetId, reportId } of assets) {
  	await this.checkAssetChecksum(originalPath, checksum, assetId, reportId);
  	processed++;
  	if (processed % 100 === 0) {
  	printStats();
  	}
  	if (Date.now() > startedAt + timeLimit || processed > count * percentageLimit) {
  	this.logger.log('Reached stop criteria.');
  	lastCreatedAt = createdAt;
  	break;
  	}

  ). That is the only thing happening at the failing instant.
• Breaking out abandons the postgres.js cursor on the reserved connection that kysely-postgres-js holds for the stream. The next job on the same queue starts in the same tick and its first query fails synchronously with CONNECTION_DESTROYED. Whether it is the same connection being handed back before the cursor is closed, or the abandoned cursor tearing the connection down after it is back in the pool, I have not pinned down. The symptom only ever hits whichever job starts at that exact moment, which is why Failed to create memories for : Error: write CONNECTION_DESTROYED database #24400 and Smart search job fails with errors: CONNECTION_DESTROYED immich_postgres:5432 #24971 look like cousins of this (same error, other scheduled jobs).

The other QueueAll jobs in the same service paginate with batched(...) rather than streaming, and the checksum job could do the same (fetch by createdAt in pages and stop between pages), which would avoid abandoning a server side cursor altogether.

The OS that Immich Server is running on

Ubuntu 24.04.5 LTS

Version of Immich Server

v3.3.1 (failures were observed on v3.1.0; the code path in handleChecksumFiles is unchanged on current main)

Version of Immich Mobile App

Not involved (server side job)

Platform with the issue

• Server
• Web
• Mobile

Device make and model

n/a

Your docker-compose.yml content

# Stock docker-compose.yml from the release, unchanged apart from pinning image tags
# (immich-server and immich-machine-learning on ${IMMICH_VERSION}, valkey 9 and
# immich-app/postgres 14-vectorchord0.4.3-pgvectors0.2.0 pinned by digest) and the
# local volume paths. No extra services, no custom networks, no resource limits.

Your .env content

UPLOAD_LOCATION=<redacted>
DB_DATA_LOCATION=<redacted>
DB_PASSWORD=<redacted>
DB_USERNAME=<redacted>
DB_DATABASE_NAME=<redacted>
IMMICH_VERSION=v3.3.1
TZ=Europe/London
MACHINE_LEARNING_MODEL_TTL=<redacted>
MACHINE_LEARNING_PRELOAD__CLIP__TEXTUAL=<redacted>

Reproduction steps

1. Library large enough that the checksum job never finishes inside integrityChecks.checksumFiles.timeLimit (default 1 h). Here roughly 140k assets at about 125 ms per asset.
2. Leave integrityChecks.untrackedFiles, missingFiles and checksumFiles on the default cron 0 3 * * *.
3. Wait for the nightly run. At exactly timeLimit after the checksum job started it logs Reached stop criteria, and in the same second IntegrityUntrackedFilesQueueAll and IntegrityMissingFilesQueueAll fail with write CONNECTION_DESTROYED <db>:5432 and a durationMs under 2 ms. It is a race: 2 nights out of 3 here.
4. Move the two report crons to a time when the checksum job is not stopping (for example 0 2 * * *). The failures stop.

Relevant log output

# Current state with the workaround in place (v3.3.1, 24 Oct). Checksum job still stops on the tick:
10/24/2026, 3:59:47 AM     LOG [Microservices:IntegrityService] Processed 28200 files so far... (avg. 127.21 ms/asset, 19.86% of all assets)
10/24/2026, 4:00:00 AM     LOG [Microservices:IntegrityService] Reached stop criteria.
10/24/2026, 4:00:00 AM     LOG [Microservices:IntegrityService] Processed 28286 files so far... (avg. 127.27 ms/asset, 19.92% of all assets)
10/24/2026, 4:00:00 AM     LOG [Microservices:IntegrityService] Finished checksum job, will continue from 2026-03-27T14:27:24.193Z.

# Report jobs now run at 02:00 and complete:
10/24/2026, 2:00:00 AM     LOG [Microservices:IntegrityService] Checking for out of date untracked file reports...
10/24/2026, 2:00:00 AM     LOG [Microservices:IntegrityService] Queued report check of 1022 report(s) (1022 so far)
10/24/2026, 2:00:00 AM     LOG [Microservices:IntegrityService] Scanning for untracked files...

# The failing lines from 28 and 30 Aug (v3.1.0) were, from my notes at the time, of the form
# "Unable to run job handler (IntegrityUntrackedFilesQueueAll): Error: write CONNECTION_DESTROYED database:5432"
# with durationMs 0.19 to 1.44 ms, logged in the same second as "Reached stop criteria".
# The container has since been recreated so I no longer have those raw lines to paste.

Additional information

Related by error signature only: #24400 (memories job) and #24971 (smart search), both closed without a root cause in the thread. I have not seen the error on any job that does not start at the checksum job's stop tick.