Skip to content

Video: normalize audio timestamps on the direct reader path - #2400

Open
dallo28 wants to merge 2 commits into
TelegramMessenger:masterfrom
dallo28:fix/direct-audio-timestamps
Open

dallo28 wants to merge 2 commits into
TelegramMessenger:masterfrom
dallo28:fix/direct-audio-timestamps

Conversation

@dallo28

@dallo28 dallo28 commented Oct 25, 2026 •

Copy link
Copy Markdown

Fixes https://bugs.telegram.org/c/66475: since 13.0, some videos play with a loud crackle whenever there is sound. The same file plays cleanly when saved to Photos, on Android and on desktop, and played fine in 12.9.3.

Cause

13.0 made ChunkMediaPlayerV2 the default for ordinary videos. In NativeVideoContent, 12.9.2 had useLegacyImplementation = !playerV2 (the debug setting defaults to false, so the legacy MediaPlayer played them). 13.0 has false. The legacy player feeds decoded audio to its own audio unit as one continuous stream. ChunkMediaPlayerV2 queues each buffer on an AVSampleBufferAudioRenderer at its presentation timestamp.

Ordinary videos go through the direct reader branch of fillRendererBuffer. The parts (HLS) branch passes audio buffers through normalizeAudioSampleBuffer (8612c4d), which makes consecutive buffers contiguous when their timestamps are within 2 ms of it. The direct branch queued the timestamps from the file as they were.

Whenever those timestamps don't advance by exactly the number of samples in the buffer, the renderer gets a gap or overlap at each boundary:

  • A 44.1 kHz track is resampled to 48 kHz. Each 1024-sample AAC frame becomes 1114.56 samples, so buffers alternate between 1114 and 1115 samples while their timestamps stay on the 44.1 kHz grid. Almost half of the boundaries are off by a fraction of a sample.
  • Some files have AAC frame durations that vary in the container (e.g. 1008 and 1056 next to 1024), although every frame decodes to 1024 samples.

Change

Pass audio buffers through normalizeAudioSampleBuffer on the direct branch too. The timing state (audioBufferTimingState) is already reset on seek and when the audio renderer is added or removed. A jump of more than 2 ms re-anchors it, as on the parts branch.

Testing

Recordings attached to the card. They were decoded and their clicks timed against the AAC frame length. The phase-lock strength runs from 0 to 1:

Recording Click period Phase lock
iPhone screen recording (app audio captured digitally) 23.22 ms = 1024 samples at 44.1 kHz 0.75
Crackling video, recorded with another phone 21.33 ms = 1024 samples at 48 kHz 0.84
Clean video, same phone none 0.09

iOS 27 simulator, debug build. I played a reporter's video (44.1 kHz AAC) in the chat's player, with a temporary log of each audio buffer's gap from the end of the previous one (not part of this change):

Build Path Buffers off by more than 0.5 samples (of 400) Largest gap
Before direct 177 16.56 samples
This change direct 0 0

The check was done through that log, not by ear, and not on a device.

Second change: jittery audio timestamps (40 ms window)

Some videos still crackled with the first commit. In those files the audio track's frame durations range from 768 to 1406 ticks (only 66% are 1024), while every frame decodes to 1024 samples. Their timestamps wander by up to ~30 ms around the decoded sample position, in steps of 2.6–8 ms. The normalizer only snapped buffers within 2 ms, so each larger step re-anchored to the file's timestamp and left a gap or overlap. Photos and Telegram Desktop play these files cleanly: Desktop reads the audio timestamp once at the first frame after a seek and then plays decoded samples back to back (AbstractAudioFFMpegLoader::enqueueNormalFrame).

Change: widen the window to 40 ms. Within 40 ms of the running sample count, buffers play back to back; beyond it the normalizer re-anchors as before. 40 ms keeps the largest possible audio/video offset under the point where audio leading video becomes noticeable.

Testing (iOS 27 simulator, debug build with a temporary log of the shift the normalizer applies):

Lesson video (~30 min, jittery stts, direct path) Discontinuous buffer boundaries Re-anchors Largest shift
First commit (2 ms) 40 in the first 151 buffers (3.2 s) at every step above 2 ms —
40 ms window 0 over 3,000 buffers 0 9.15 ms

From the file's stts, its timestamps drift at most 29.1 ms between any two points, so it can't cross 40 ms even after a seek. Checked through the log, not by ear, and not on a device.

Known limit (HLS): each HLS part loses the few samples still held by its resampler at the end of the part (~16 per part when 44.1 kHz audio is played at 48 kHz), because FFMpegAudioFrameDecoder.sendEndToDecoder() does nothing. With the 2 ms window that showed up as a small re-anchor about every 30 s; with 40 ms the loss builds up to ~36 ms over ~9 minutes before a re-anchor. Draining the resampler at the end of each part fixed that in testing (flat at ~0.37 ms over 21,000 buffers), but it is left out of this PR: with it, an HLS video played to its end and then restarted from the beginning stopped producing audio after a few seconds, and it wasn't confirmed whether the drain caused that. It is left for a follow-up.

Since 13.0 ordinary videos play in ChunkMediaPlayerV2 instead of the
legacy MediaPlayer. They go through the direct reader branch of
fillRendererBuffer, which queued every audio buffer at the timestamp
read from the file. The parts (HLS) branch already passes them through
normalizeAudioSampleBuffer; the direct branch did not.

When the timestamps don't step by exactly the buffer's sample count,
AVSampleBufferAudioRenderer gets a gap or an overlap at the buffer
boundary, heard as a click. With a 44.1 kHz track resampled to 48 kHz,
buffers alternate between 1114 and 1115 samples while their timestamps
stay on the 44.1 kHz grid, so almost half of the boundaries are off by
a fraction of a sample: a constant crackle whenever there is sound.
Files whose frame durations vary (1008 and 1056 next to 1024) do the
same.

Normalize audio buffers on the direct branch too. The timing state is
already shared and reset on seek and renderer changes.

Fixes https://bugs.telegram.org/c/66475
Some videos still crackled after audio timestamps were normalized on the
direct reader path. Their audio tracks list frame durations between 768
and 1406 ticks while every AAC frame decodes to 1024 samples, so the
timestamps wander by up to ~30 ms around the decoded sample position.
normalizeAudioSampleBuffer only snapped buffers within 2 ms of the
running sample count, so each larger step re-anchored to the file's
timestamp and left a gap or overlap at the boundary. Telegram Desktop
anchors once and plays decoded samples back to back, so it plays these
files cleanly.

Widen the snap window to 40 ms. Within it buffers play back to back;
beyond it the normalizer re-anchors as before. 40 ms keeps the largest
possible audio/video offset under the point where audio leading video
becomes noticeable.

Fixes https://bugs.telegram.org/c/66475
@dallo28
dallo28 force-pushed the fix/direct-audio-timestamps branch from ec9eed4 to 2f2c807 Compare October 30, 2026 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant