Repository navigation
Drop peer read state sync operations for invalid PeerId(0) (infinite retry busy loop) - #2418
Open
igabdullin wants to merge 3 commits into
Open
igabdullin wants to merge 3 commits into
igabdullin wants to merge 3 commits into
Conversation
A garbage row for PeerId(0) (Push(state: nil, thenSync: true)) in the peer read state sync queue makes SynchronizePeerReadStatesContextImpl restart an operation that can never succeed (peer 0 cannot resolve to an Api.InputPeer), 830+ times per second, pinning ~1.1 CPU cores indefinitely. The manager retries failed operations immediately with no backoff, so a permanently-failing operation becomes a busy loop that also storms the postbox queue with transactions and view invalidations. Clear such rows via confirmSynchronizedIncomingReadState, the same way the !keepsReadState path clears non-member channels.
A zero peer id is the minimum sentinel and never a real peer. The read state table currently accepts an operation for it: an interactive read applied with a zero-peer index manufactures Push(state: nil, thenSync: true) for a peer that has no read state, and once queued the operation can never complete on the consumer side (it cannot resolve to an InputPeer) and is retried without backoff forever, pinning a core (telegram-macos#194). Treat any attempted operation for the zero peer id as a removal at the table chokepoint, covering every writer; a valid stateless peer keeps today's behavior.
Three cases: an interactive read with a zero-peer index enqueues nothing (red on unfixed master), a valid stateless peer still enqueues Push(state: nil, thenSync: true), and loadedPeerWithId of a missing row never emits (documents why an instantly retrying loop implies a stored peer row).
igabdullin
force-pushed
the
fix/peer-id-zero-readstate-loop
branch
from
October 30, 2026 16:20
64c6024 to
fe97b6e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the CPU burn described in TelegramMessenger/telegram-macos#194 (root-cause analysis, the
wider class of never-completing operations, and a proposed backoff layer in the follow-up
comments there).
Two layers, one per commit.
Commit 1 — chokepoint: never queue an operation for the zero peer id
MessageHistorySynchronizeReadStateTable.setnow treats any attempted operation for a peerwhose id is
PeerId.Id.min(zero — the minimum sentinel, never a real peer) as a removal.Why the chokepoint: a zero peer id is not a real peer, and an operation queued for it can never
complete on the consumer side — it cannot resolve to an
Api.InputPeer, so the manager retriesit without backoff forever (the incident: 831 retries/sec, ~1.1 cores for 8+ hours, survives
restarts and network disconnect). Rejecting at
set()covers every writer —Pushwith anythenSync, andValidate(e.g. the message-deletion and hole-invalidation paths) — and alsoclears a stored row whenever any writer touches that peer again.
Any valid peer, stateless included, keeps today's behavior (a stateless peer still gets
Push(state: nil, thenSync: true)— that path is legitimate and stays).Commit 2 — consumer cleanup: rows already persisted in users' databases
The chokepoint stops new rows, but users hitting this bug today already have the zero-peer row
persisted — and nothing guarantees anyone will call
set()for that peer again, so thechokepoint alone may never rewrite it. This is why the manager-side guard stays:
ManagedSynchronizePeerReadStatesContextImpl.updatedrops and clears a zero-peer operation viaconfirmSynchronizedIncomingReadStateon the first pass — the same way the!keepsReadStatepath clears non-member channels.
Regression tests (commit 3)
Tests/GhostPeerZeroReadStateOpTests.swift— picked up automatically by thePostboxTestsBazel target (
srcs = glob(["Tests/**/*.swift"])). Three cases:Push(thenSync: true)— guards againstover-blocking;
loadedPeerWithIdof a missing row never emits (documents why the incident's instantretries imply a stored peer row).
Local verification (out-of-tree harness against the Postbox sources at fb0e62b)
applyInteractiveReadMaxIndexwith a zero-peer index →0:Id(rawValue: 0) -> .Push(state: nil, thenSync: true)appears insynchronizePeerReadStatesView()— byte-for-byte the row observed live in the incident log;PeerId(0);loadedPeerWithIdsemantics unchanged.Not included (discussed in the linked issue)
(e.g. a stored peer without accessHash). Concrete diff in the issue; kept out to keep this PR
minimal and provably behavior-preserving.
peerKeepsReadState(nil) == falseand dropping operations for stored peers withoutaccessHash— both change semantics for legitimate flows; maintainer's call.