Skip to content

Video: convert any audio layout, preferring an AAC track - #2396

Open
dallo28 wants to merge 1 commit into
TelegramMessenger:masterfrom
dallo28:fix/video-apac-audio-track
Open

dallo28 wants to merge 1 commit into
TelegramMessenger:masterfrom
dallo28:fix/video-apac-audio-track

Conversation

@dallo28

@dallo28 dallo28 commented Oct 24, 2026 •

Copy link
Copy Markdown

Fixes https://bugs.telegram.org/c/67051 and https://bugs.telegram.org/c/67049: some iPhone videos fail to send from the attachment menu. The message gets a red error mark at once while the "Processing" ring keeps spinning.

Cause

With "Record Sound: Spatial Audio", the Camera app writes an APAC track (Apple Positional Audio, 4-channel first-order ambisonics) next to an AAC stereo track. In the original file the AAC track comes first. Trimming the video in Photos rewrites it with the APAC track first and enabled, and the AAC track disabled:

Original (sends) Trimmed in Photos (fails)
1st audio track AAC stereo, enabled APAC 4 ch, enabled
2nd audio track APAC 4 ch, disabled AAC stereo, disabled
metadata tracks 6 1

TGMediaVideoConverter (unedited sends) and MediaEditorVideoExport (edited sends, stories) both take the first audio track and decode it with no channel constraint. The AAC writer input refuses the first buffer whose channel count differs from its own (AVFoundationErrorDomain -11800, OSStatus -12780), so the conversion fails before any upload starts. HDR isn't involved: the video track is the same HLG HEVC in both files.

The same failure hits more than Spatial Audio. Testing every layout in the app showed it for 4- and 9-channel ambisonic PCM (ProRes Spatial Audio), B-format, quad and 5.1 tracks, and for plain stereo PCM whenever the quality preset encodes mono audio (AAC sources happen to be downmixed by the decoder, PCM ones are not).

Change

  • Track: take an AAC track when the file has one, otherwise the first audio track as before. For Spatial Audio videos this is the stereo mix the camera recorded, and it skips rendering the APAC track (about 60 ms of CPU per second of audio). _preferredAudioTrackForAVAsset: in TGMediaVideoConverter, preferredAudioTrack(in:) in MediaEditorUtils.swift (the main video and collage/dual-video inputs; the added-music track is unchanged).
  • Decoding: the audio reader decodes straight to the encoder's channel count and layout (from audioSettingsForPreset: / configuration.audioSettings), so Core Audio downmixes or renders whatever track is used: APAC and PCM ambisonics (ACN SN3D/N3D, B-format) are rendered as a sound field (a source on the left stays on the left), 5.1 and quad are downmixed, stereo is folded to mono when the preset is mono. Mono and stereo AAC sources come out identical to before (same channels, length and level). The channel count stays the preset's, as for every other video: stereo at Medium and above (so a camera video's APAC is rendered to stereo), mono at Low and Very Low, none for animations and profile videos.
  • Logging: FetchVideoMediaResource logs conversion and export errors (library and local-file paths, both pipelines) under FetchVideoResource; the legacy converter reports the writer's error when the reader has none (it reported nil before) and logs an exception raised by appendSampleBuffer.

"Send as file" still uploads the original file untouched, including the APAC track.

Logic

Both converters now handle audio the same way:

  1. Pick the track. The first AAC track in the file, even if it is disabled (Photos disables it in trimmed Spatial Audio videos). With no AAC track, the first audio track, whatever its format.

  2. Decode to the encoder's format. The reader outputs linear PCM with the channel count and layout of the AAC encoder, which come from the quality preset:

    Preset Encoder What happens to the source audio
    Medium, High, Very High AAC stereo, 44.1 kHz AAC/PCM stereo passes through; mono is duplicated to both channels; APAC and PCM ambisonics are rendered to stereo as a sound field; 5.1 and quad are downmixed
    Low, Very Low AAC mono, 44.1 kHz everything is folded to mono (ambisonics rendered first)
    Animation, profile no audio audio is dropped, as before
  3. Encode with the same AAC settings as before. Sample-rate conversion stays with the encoder, as before.

In MediaEditorVideoExport the same settings go to the AVAssetReaderAudioMixOutput, so the editor's volume mix and an added music track are mixed into that same layout. When the encoder settings have no channel count (stickers, avatars), the reader settings stay nil, as before.

Testing

Built with Make.py build --configuration=debug_sim_arm64 and run in the iOS 27.0 simulator (iPhone 17e), logged in, sending each file on its own to Saved Messages from the attachment menu, then checking the app log and the converted file in the media cache:

File Audio tracks Before After
Reporter's file trimmed in Photos APAC first, AAC (disabled) fails sent, from the AAC track
Reporter's original file AAC first, APAC sent sent, unchanged
t14 APAC first, AAC stereo fails sent, from the AAC track
t01 APAC only fails sent, APAC rendered
t05 APAC 9 ch (2nd order) fails sent
t02b PCM 4 ch ambisonics, PCM stereo (ProRes layout) fails sent
t10 PCM 4 ch ambisonics (N3D) fails sent
t03 PCM B-format fails sent
t04 PCM 9 ch ambisonics fails sent
t11 PCM quad fails sent
t06 AAC 5.1 fails sent
t09 PCM stereo (mono preset) fails sent
t07 AAC mono sent sent
t08 no audio sent sent

Edited sends (trimmed in Telegram's editor, MediaEditorVideoExport): the reporter's trimmed file (AAC track used, L/R correlation 0.99 like the camera's AAC) and t10 (left-panned tone rendered left: L −10.0 dBFS, R −27.3 dBFS) both sent.

Which track was used was checked on the output: the reporter's file now matches its AAC track (L/R correlation 0.99; the rendered APAC is 0.95), and t14, whose AAC track carries the tone in the left channel only, comes out like that AAC track rather than like the rendered APAC. Every output keeps the source length (minus the 0.1 s the legacy converter always trims) and level.

Not tested on a device. Please send a Photos-trimmed Spatial Audio video from the attachment menu on an iPhone 16 or 17.

Limitations

  • If Photos stores an "Audio Mix" edit (Cinematic/Studio) only as metadata for playback, neither the AAC track nor the rendered APAC includes it. Not verified.
  • When a conversion does fail, the message still shows the "Processing" ring until the chat is closed (FetchManagerImpl has no error handler); an album with one failing item waits for the rest, then sends without it. The new log lines at least show why.
@dallo28
dallo28 force-pushed the fix/video-apac-audio-track branch from 46c0a7e to 64084a7 Compare October 24, 2026 21:29
@dallo28 dallo28 changed the title Video: don't convert from a leading APAC audio track Oct 24, 2026
Videos recorded with Spatial Audio carry an APAC (ambisonic) track next
to an AAC stereo track, and trimming one in Photos puts the APAC track
first. Both the legacy converter and MediaEditorVideoExport took the
first audio track and fed it, decoded as is, to the AAC encoder, which
refuses any channel count other than its own (AVFoundationErrorDomain
-11800, OSStatus -12780). Sending failed at once: the message got a red
error mark while the "Processing" ring kept spinning. The same failure
hit PCM audio (ProRes recordings, including stereo PCM when the preset
encodes mono), 5.1 and other multichannel tracks.

- Take an AAC track when the file has one, which skips rendering the
  APAC track and is what the camera mixed for stereo playback.
- Decode audio straight to the encoder's channel count and layout, so
  Core Audio downmixes or renders whatever track is used: APAC and PCM
  ambisonics (ACN SN3D/N3D, B-format), 5.1, quad, stereo PCM.
- Log conversion errors in FetchVideoMediaResource, and report the
  writer's error when the reader has none.

Fixes https://bugs.telegram.org/c/67051 and https://bugs.telegram.org/c/67049
@dallo28
dallo28 force-pushed the fix/video-apac-audio-track branch from 64084a7 to 98ac708 Compare October 24, 2026 22:22
@dallo28 dallo28 changed the title Video: render ambisonic audio to stereo when converting Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant