On April 1 at 10 PM I typed “NO DEMO IF THE SOUND SUCKS”. The demo film for Immich Memories, my tool for turning an Immich photo library into short films, had its sound and picture drifting apart. Every source clip played fine. The assembled MP4 decoded without a single error. It just wasn’t watchable.

Six days later the commit went in, along with my summary of the week: “6 days to fix this freaking issue”. Then in September the same drift came back, through a door the April fix had left open. This is both rounds.

Two pipelines, one edit

The assembler decodes video frames into Python, blends the crossfades and streams the result into one FFmpeg encoder (so memory stays flat no matter how many clips the film has). Audio goes a different way: extracted per clip, normalised, crossfaded, then muxed with the finished video.

That keeps decoded video out of the audio mix, but it means the edit’s timing exists twice: once in the frame loop, once in an FFmpeg audio graph. Every place they disagree about a clip’s length or a fade’s length shows up as drift. And since the error per clip is milliseconds, you only see it on a long film.

Six days of guessing

The direction kept flipping, which is the first sign you’re fixing symptoms. April 1: “audio starts roughly 1, 1.5s late”. April 2 evening: “always around 2 seconds”, with the sound early this time. April 3 morning: “okay we’re back on the REVERSE, now it’s TOO LATE (the stuff we fixed)”.

Claude’s theories that week (most of its April replies are lost, but my prompts survive with timestamps, and they tell the story):

• Clip selection. My answer: “OF COURSE IT’S NOT CLIP SELECTION”.
• Silent title cards bleeding into the next clip through the audio crossfade. The fix concatenated audio at title boundaries instead. It didn’t fix anything and got reverted the same morning.
• A crossfade explanation for a specific bad moment. My answer: “no no no ! the frame 139 is well into the clip, it’s not during the crossfade.”

My side of the conversation, roughly in order: “can you COMPARE THE CLIPS audio VS the FINAL AUDIO ?”, “LOOK. AT. THE. VIDEO OUTPUT.”, “don’t guess”, “STOP GUESSING”. One of my guesses was right, for once: “i think it’s the TITLES that causes issues”, followed eight minutes later by “TITLE SCREENS SHOULD BE RENDERED AFTER MAX FPS DETECTION HAS BEEN DONE”. Title cards were taking part in detecting the film’s frame rate. Excluding them was one of the real fixes.

On April 3 a short test passed: “YES YES YES IT SEEMS PERFECT. do this for the full year”. Ten hours later: “nope. … still the same problem around 1:04”. Short films hide drift. On April 4 the session had gone in circles long enough that I asked Claude to write its own restart prompt: “I think your context is rottent, can you provide a prompt to restart this investigation”. April 5: “SEARCH ONLINE. BE METHODIC”.

Counting frames by hand

What ended the guessing was the least clever thing in this post. On April 7 I asked: “Is there a way to give you the EXACT frame when my mouth move + the exact one where the sound goes out ?” Then I stepped through the film in VLC, frame by frame, and wrote the numbers down:

Sam, Apr 7 09:29 (VLC):  sound : frame 0113. lips starts to move at frame 0135 (mouth opening)
Sam, Apr 7 11:36/11:44:  Frame 139 is the one where the sound starts. ... it's actually frame 151 !

22 frames at 30 fps is 0.73 seconds of sound before the lips move. A frame number is a measurement. “It feels about two seconds late” is not, and we’d spent five days on the second kind.

What was actually wrong in April

Four things, each small, all multiplied by clip count.

Clip lengths. The video loop writes whole frames: int(duration * fps). The audio used the requested duration. A 3.55-second clip at 30 fps is 106.5 frames: the video gets 106 (3.5333 s), the audio keeps 3.55 s. That’s 800 samples at 48 kHz, every clip with an awkward length. The audio graph now uses the duration the frames actually represent:

frame_duration = int(clip.duration * fps) / fps

Normalisation. Loudness normalisation sat between extraction and mixing, and nothing guaranteed it returned exactly the duration it was given. Each segment is now trimmed to its frame-derived length before and after normalising, with timestamps reset (atrim and asetpts are separate jobs in FFmpeg: trimming samples doesn’t reset their timestamps):

%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
flowchart TB
    format["Set format and sample rate<br/>Reset timestamps"] --> bound["Pad / trim each segment<br/>to frame-derived duration"]
    bound --> normalize["Normalize"] --> trim["Trim again"]
    trim --> joins["Crossfade / concatenate<br/>following the video's transition list"]
    joins --> final["Pad / trim soundtrack<br/>to finished video duration"]

Transitions. The audio graph now follows the video’s transition list exactly: acrossfade where the video crossfades, concat where it cuts, silence of the right length for title cards.

One AAC encode. At one point the drift was “fixed” with a hard-coded offset. My quote of Claude’s claim back to it: “-itsoffset 0.2 compensates perfectly”. It compensated on my machine. “ONE THING we need to avoid is having something that works fine FOR ME. but does not for others”. The real fix kept everything as PCM WAV through the mix and encoded AAC exactly once, then muxed with -c:a copy. The commit called it “Hardware-independent”, which was the point.

Two tests came with it that I still like: one runs 30 clips through real FFmpeg normalisation and asserts the total duration stays within 50 ms, and one asserts the audio graph trims to atrim=0:3.0 and not 3.017. The pull request’s result line: “Full year test (70 clips, 2023) — 1-2 frame offset (33-66ms)”. Somewhere in the middle of that morning, my message at 10:50: “OH YES YES YES CLAUDE sometimes you suck, but most of the time you’re amazing”.

The hang after the last frame

That evening the long render reached 100% and never finished: “okay it seems GOOD but once again… we lost the ffmpeg process and it failed”.

Claude’s first diagnosis was a leaked file descriptor: the last decoder’s FFmpeg child holding the encoder’s stdin pipe open, so the encoder never saw end-of-file. It added explicit closes. The re-run: “okay again, 100% encoding but seems to be stuck/dead…”.

Then it did the right thing and looked at the live processes instead of re-arguing the theory. Its report:

Same problem — my fix didn’t work. The FD inheritance theory from the comment might be wrong. Only the encoder itself holds the pipe — no leaked FDs. The previous diagnosis was completely wrong.

The real cause was a classic. The encoder ran with stderr=PIPE, and the code called Popen.wait() without reading stderr. FFmpeg’s progress output filled the 64 KB pipe buffer, FFmpeg blocked writing to it, and Python waited forever for a process that was waiting for Python (the subprocess docs warn about exactly this). Draining stderr before waiting fixed the hang at the end of the encode (confirmed that night on a 60-second film).

Two footnotes. The comment describing the wrong theory survived in the codebase for months, because the squash merge kept it. And draining at the end didn’t cover long encodes, where the buffer fills during the write; that got its own fix in August, a thread draining stderr from the first frame.

September: same drift, different door

In September, a seven-minute film with 0.25-second crossfades came out with its audio 1.3 seconds ahead by the end. Codex measured the lead at four points:

| Final position | Audio lead | Approx. fades after fixed ~2-frame baseline |
| 78.73 s        | 0.2334 s   | 10    |
| 254.77 s       | 0.7668 s   | 42    |
| 404.17 s       | 1.2229 s   | 69–70 |
| 414.47 s       | 1.3003 s   | 74    |

A straight line against the number of fades. Its first move was to sidestep the problem: “use a music-only soundtrack so the known source-audio drift cannot spoil the ending”. Muting everyone’s voices to hide a sync bug. My answer started with a swear word and ended with “LOOK IN THE GIT HISTORY we WORKED and WORKED and WORKED on that”.

After that it did good work. It ruled out normalisation by counting samples (zero lag through the April trim sandwich), built a two-clip synthetic proof, and found the gap April had left: the clip lengths were frame-derived, the fade wasn’t.

video: int(0.25 × 30) = 7 frames = 0.233333… s
audio: acrossfade=d=0.25 = 0.250000 s
difference: 0.016666… s = 800 samples at 48 kHz per fade
74 fades: 1.233333… s
measured final lead in the defective film: approximately 1.300 s

The video fade rounds down to 7 whole frames. The audio crossfade overlaps the full 0.25 seconds. Each fade, the audio gets 800 samples shorter than the picture, and everything after it plays early. At the default 0.5-second fade, 0.5 × 30 is exactly 15 frames, which is why April’s 70-clip test passed.

I reproduced it while writing this, on the current code, with no footage at all: 40 synthetic clips, each with a white flash and a beep at the same instant, through the production assembler, then every flash matched to its beep in the output:

16.7 ms per fade, exactly the 800 samples. With both fades set to the same whole-frame length, the line goes flat. (The constant +14 ms is the same on every clip, so it’s a fixed offset in my measurement, not drift.)

The September film itself was re-rendered that day, with the residual down to 0, 33 and 67 ms at the three probe points. The last frame or two most likely came from decoders silently delivering one frame fewer than planned (“one decoder expecting 111 frames and producing 110”).

What shipped

• April: frame-derived clip lengths, trims around normalisation, audio transitions that follow the video’s list, title cards out of frame-rate detection, one AAC encode. The 70-clip year film landed within one to two frames.
• The hang: stderr drained before the wait in April, then on a thread from the first frame in August.
• September: the audio crossfade gets the same whole-frame duration as the video fade. On the clapper test, that’s the flat line.

What the models did, and didn’t

April was Claude Code, September was Codex. Both followed the same arc: confident guesses first, measurement only when pushed.

The useful parts were real. Claude wrote the commit that explained the cumulative April causes, and the at-scale FFmpeg tests. When its file-descriptor theory failed, it checked the live processes and retracted it in one message, which is more than a lot of people manage. Codex’s sample-count proof in September was exactly the right method.

The rest: a week of theories with no measurement behind them, a hard-coded offset that worked “FOR ME”, a confident diagnosis of the hang that was wrong, and a proposal to mute clip audio rather than fix it. None of those were caught by a test. They were caught by me watching the film, and then, finally, by me counting frames in VLC and handing over numbers instead of impressions.

If you’re debugging A/V sync: put a flash and a beep in every clip, and measure frames. It’s the clapperboard, and it’s been the answer since the 1920s.