Skip to content

Drop peer read state sync operations for invalid PeerId(0) (infinite retry busy loop) - #2418

Open
igabdullin wants to merge 3 commits into
TelegramMessenger:masterfrom
igabdullin:fix/peer-id-zero-readstate-loop
Open

igabdullin wants to merge 3 commits into
TelegramMessenger:masterfrom
igabdullin:fix/peer-id-zero-readstate-loop

Conversation

@igabdullin

@igabdullin igabdullin commented Oct 29, 2026 •

Copy link
Copy Markdown

Fixes the CPU burn described in TelegramMessenger/telegram-macos#194 (root-cause analysis, the
wider class of never-completing operations, and a proposed backoff layer in the follow-up
comments there).

Two layers, one per commit.

Commit 1 — chokepoint: never queue an operation for the zero peer id

MessageHistorySynchronizeReadStateTable.set now treats any attempted operation for a peer
whose id is PeerId.Id.min (zero — the minimum sentinel, never a real peer) as a removal.

Why the chokepoint: a zero peer id is not a real peer, and an operation queued for it can never
complete on the consumer side — it cannot resolve to an Api.InputPeer, so the manager retries
it without backoff forever (the incident: 831 retries/sec, ~1.1 cores for 8+ hours, survives
restarts and network disconnect). Rejecting at set() covers every writer — Push with any
thenSync, and Validate (e.g. the message-deletion and hole-invalidation paths) — and also
clears a stored row whenever any writer touches that peer again.

Any valid peer, stateless included, keeps today's behavior (a stateless peer still gets
Push(state: nil, thenSync: true) — that path is legitimate and stays).

Commit 2 — consumer cleanup: rows already persisted in users' databases

The chokepoint stops new rows, but users hitting this bug today already have the zero-peer row
persisted — and nothing guarantees anyone will call set() for that peer again, so the
chokepoint alone may never rewrite it. This is why the manager-side guard stays:
ManagedSynchronizePeerReadStatesContextImpl.update drops and clears a zero-peer operation via
confirmSynchronizedIncomingReadState on the first pass — the same way the !keepsReadState
path clears non-member channels.

Regression tests (commit 3)

Tests/GhostPeerZeroReadStateOpTests.swift — picked up automatically by the PostboxTests
Bazel target (srcs = glob(["Tests/**/*.swift"])). Three cases:

  • zero-peer interactive read enqueues nothing (red on master, green with commit 1);
  • a valid stateless peer still enqueues Push(thenSync: true) — guards against
    over-blocking;
  • loadedPeerWithId of a missing row never emits (documents why the incident's instant
    retries imply a stored peer row).

Local verification (out-of-tree harness against the Postbox sources at fb0e62b)

  • before: applyInteractiveReadMaxIndex with a zero-peer index →
    0:Id(rawValue: 0) -> .Push(state: nil, thenSync: true) appears in
    synchronizePeerReadStatesView() — byte-for-byte the row observed live in the incident log;
  • after: no operation is created for PeerId(0);
  • no behavior change for valid stateless peers; loadedPeerWithId semantics unchanged.

Not included (discussed in the linked issue)

  • Retry backoff in the manager — bounds damage from any other never-completing operation
    (e.g. a stored peer without accessHash). Concrete diff in the issue; kept out to keep this PR
    minimal and provably behavior-preserving.
  • peerKeepsReadState(nil) == false and dropping operations for stored peers without
    accessHash — both change semantics for legitimate flows; maintainer's call.
A garbage row for PeerId(0) (Push(state: nil, thenSync: true)) in the peer
read state sync queue makes SynchronizePeerReadStatesContextImpl restart
an operation that can never succeed (peer 0 cannot resolve to an
Api.InputPeer), 830+ times per second, pinning ~1.1 CPU cores
indefinitely. The manager retries failed operations immediately with no
backoff, so a permanently-failing operation becomes a busy loop that
also storms the postbox queue with transactions and view invalidations.

Clear such rows via confirmSynchronizedIncomingReadState, the same way
the !keepsReadState path clears non-member channels.
A zero peer id is the minimum sentinel and never a real peer. The read
state table currently accepts an operation for it: an interactive read
applied with a zero-peer index manufactures Push(state: nil, thenSync:
true) for a peer that has no read state, and once queued the operation
can never complete on the consumer side (it cannot resolve to an
InputPeer) and is retried without backoff forever, pinning a core
(telegram-macos#194). Treat any attempted operation for the zero peer id
as a removal at the table chokepoint, covering every writer; a valid
stateless peer keeps today's behavior.
Three cases: an interactive read with a zero-peer index enqueues nothing
(red on unfixed master), a valid stateless peer still enqueues
Push(state: nil, thenSync: true), and loadedPeerWithId of a missing row
never emits (documents why an instantly retrying loop implies a stored
peer row).
@igabdullin
igabdullin force-pushed the fix/peer-id-zero-readstate-loop branch from 64c6024 to fe97b6e Compare October 30, 2026 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant