The bike race demo on my project’s docs page replayed the same 0.21 seconds of a cyclist twice. It had been up there since March, under a heading saying the stitching worked. Left is the version I published in March, right is the one I rebuilt in September:

Six frames is easy to miss at full speed (the clip slows the join down to 0.25× at the end). It’s a lot less easy to miss when it’s a race car, or your kid blowing out candles, going backwards for a beat.

This is Immich Memories, my tool for turning an Immich photo library into short films. When you take three or four Live Photos in a few seconds, their companion videos overlap: each one is ~3 seconds of footage around the shutter press. Stitched properly, a burst becomes one longer continuous shot. Stitched badly, it jumps back and forth at every join.

The fix ended up being “measure the overlap from the footage, and refuse when you can’t”. Getting there took one wrong method in March, a silent fallback, a guard that refused correct answers, a fallback that made things worse, and two cheap models in opencode that kept getting it almost right.

March: “Audio is perfect”

The first version was a six-hour session in March, with Claude. My notes from that day list six attempts: shutter timestamps, raw audio cross-correlation (failed on music with a beat), STFT spectrograms (worked), frame correlation (failed on repetitive scenes: kids around a Christmas tree all look alike), audio for one track and frames for the other (frames wrong on 10 of 25 pairs), and finally spectrogram alignment for both tracks.

The verdict in the notes: “Audio is perfect. Video has negligible ~50ms offset.” The bike race was “almost perfect”. I even wrote down a blog angle: merging iPhone Live Photos better than Apple. Glad I didn’t publish that one.

Frame matching went in the bin because it couldn’t handle repetitive content. Remember that, it comes back.

September: every check green, the video jumps

On September 15 I was watching a batch of 60-second control films, rendered to validate a release. A Live Photo burst from a race jumped. My note to Codex was: “the live stitch on the race is not properly aligned it jumps back and forth”.

Codex opened an issue and wrote down the uncomfortable part: the film had passed its duration checks, its source-bounds checks and its speech checks. Nothing checked visual continuity. The only test that caught it was a person watching.

The first surprise was that the March alignment wasn’t running at all. The newer, “certified” render path skipped it entirely and used the planner’s trims as-is. The older path needed SciPy for the spectrogram, and SciPy wasn’t a project dependency: without it, the alignment step fell back to the planner, silently. So the March fix had been protecting nothing for a while, and nobody (me included) noticed because the output still had the right duration.

The plan was right. The model under it wasn’t

The planner places each companion video on a shared timeline using the photo’s capture timestamp, assuming the shutter press sits at the midpoint of its video. The saved plan for the race burst:

"trim_points": [[0.0, 2.2880000000000003], [1.2999999999999998, 2.545], [0.8834999999999997, 1.767]]
"seconds": 4.4165,

The model’s first theory (GLM, more on that below) was a gap and a rewind under a “video starts 1.5 s before the shutter” model. Wrong model: the code uses the midpoint. Re-run with the code’s actual assumption, the plan was perfectly self-consistent. It then swept every cached plan from two years of renders:

25 run plans found
0 discontinuous bursts found (code model, full source durations)

Zero. The arithmetic was fine. The assumption under it wasn’t. To prove that, it matched each still photo against the frames of its own companion video, to find where the shutter actually landed inside the file:

member 1: real pre-shutter lead 0.867s vs midpoint 1.200s (error -0.333s)
member 2: real pre-shutter lead 1.567s vs midpoint 1.300s (error +0.267s)
member 3: real pre-shutter lead 1.700s vs midpoint 0.883s (error +0.817s)

Errors from −0.33 to +0.82 seconds, and what you see at a join is the difference between two files’ errors. The plan rewound 0.39 s at the first join and 0.83 s at the second, while being arithmetically perfect. The midpoint was a guess, and the code’s own docstring said so (“the midpoint alignment remains an estimate”). I’d just never measured how bad the estimate was.

Measuring the overlap from the footage

Consecutive companions overlap by construction, so the files themselves say how their clocks relate. If file B started 0.6 s after file A, then B at local time zero shows the same instant as A at 0.6 s:

A(t + offset) and B(t) show the same instant

The first measurement scored every candidate offset by mean absolute difference over tiny grayscale frames (30 fps, 64×36). On the race burst, both joins had one clear winner:

join1 on frame-differences (motion):
  offset +0.600s  mad 0.21
  offset +0.633s  mad 1.96
join2 on frame-differences (motion):
  offset +0.833s  mad 0.23
  offset +0.800s  mad 1.98

The plan had claimed start deltas of 0.988 s and 1.662 s. The footage said 0.600 and 0.833.

The shipped matcher works at 15 fps and 36 pixels tall, keeping the source’s aspect ratio (that last part came from me noticing a portrait and a landscape clip being “matched”: they can’t share a frame, so different shapes now refuse outright). It compares frame-to-frame changes rather than raw pixels, which cancels most of the static scene and exposure differences and keeps the pattern of movement:

motion_a = np.abs(np.diff(frames_a, axis=0))
motion_b = np.abs(np.diff(frames_b, axis=0))
error_by_offset = []
for d in range(len(motion_a) - 2):
    n = min(len(motion_b), len(motion_a) - d)
    if n < 3:
        break
    error_by_offset.append((float(np.abs(motion_b[:n] - motion_a[d : d + n]).mean()), d))

One probe frame is 66.7 ms. Coarse compared to the source video, but the rewinds it fixes are 6 to 12 times bigger than that.

To show the mechanism without my family’s footage, here’s a synthetic burst: one CC0 photo (“Blue stream with a waterfall”, Steve Bittinger), a fake hand-held pan, cut into three overlapping clips. Top is a plan with midpoint-style errors, bottom uses the production matcher (which recovers the true 0.6 s and 0.8 s offsets from the encoded H.264 files):

The hard part is refusing

Picking the minimum is easy. Every pair of files has a lowest score, including two files that share nothing. That’s exactly why frame matching failed in March: on repetitive content the minimum was confidently wrong, and the code believed it.

So the September matcher has three ways to say no:

• the best score must be under an absolute ceiling (12 on 0–255 grayscale differences),
• it must beat 0.7× the median candidate, so a static scene where every offset scores the same says nothing,
• it must beat every offset more than four probe frames away by 1.25×, so content that repeats itself doesn’t pass.

Here’s what that looks like on three synthetic pairs built from the same CC0 photos: a real overlap, a tripod shot where nothing moves, and two unrelated clips.

The third rule took two attempts. The first version compared the winner to the runner-up, but the runner-up is always the neighbouring offset (adjacent offsets share most of their frames), so the guard refused everything. The second version compared against offsets further out, but as a gap from the median, and it refused the Italian hilltop demo, which had a correct answer: the motion said 1.867 s and the March audio method said 1.87. Making it a ratio to the best score fixed it.

I re-ran both methods on the three published demo bursts while writing this. The motion matcher and the audio fingerprint never measured the same thing, and they agree to within one probe frame on every join the matcher accepts:

burst            motion (s)                        audio (s)
bike_race        0.600 1.267  refused 0.600 0.733  0.603 1.269 2.624 0.603 0.699
italian_hilltop  1.867 0.600                       1.867 0.603
landscape        2.133 1.200                       2.133 1.200

The one refusal is the right call. Photo 4 of the bike race starts 2.62 s into photo 3, which is only 2.87 s long: a quarter of a second of shared footage, too short to trust. It’s also the join where the March demo replayed the cyclist.

The fallback that made it worse

When a join couldn’t be measured, the first implementation fell back to the midpoint guess for that join. The model’s description: “strictly better than pure metadata”. It wasn’t. In the review batch, one four-photo burst had its middle join go from 0.00 s of rewind to 0.80 s. A guessed join sitting next to measured ones doesn’t agree with them about where the shared clock is.

Then it put the whole measured alignment behind an off-by-default flag. I overruled that about three minutes later, with two messages that became the rules: “the live stitch alignement should ALWAYS be measured” and “the midpoint guess is not good”.

So the production behaviour is binary. Every join measured: stitch. Any join unmeasurable: no stitched motion for that burst, the photos stay photos. A still is boring. A video that rewinds looks broken.

Re-placing the cuts

With measured offsets, each file gets a position on the first file’s clock, and a continuous join between files A and B needs:

end_A = start_B + delta

The end of the last file is the end of the continuous span. For the race burst, that was 3.2 seconds. The plan had asked for 4.42. The missing 1.2 seconds had never existed: it was the repeated footage.

aligned_trims keeps the planned window lengths where the footage allows, shrinks them all proportionally where it doesn’t, and returns nothing if the measured clocks would push a window before its file starts. On every measured case, each still’s own moment stayed inside its window, so shortening the shot never cut away the photo it came from.

Same idea on real footage, a chairlift burst from the docs: left plays the three companions back to back, right uses the measured offsets.

30 bursts, a trailing comma and a broken review tool

One burst isn’t a test. I asked for 30 candidates from 2024, rendered with the fix, analysed, then reviewed by eye. Some rows from that report (locations removed):

| burst               | members | plan s | aligned s | metadata join rewinds | aligned join rewinds |
| 2024-08-27-4f290e66 | 2       | 3.726  | 3.2       | 0.13s                 | 0.00s                |
| 2024-09-08-24c73976 | 2       | 3.329  | 2.4       | 0.53s                 | 0.00s                |
| 2024-08-19-f8791a76 | 2       | 3.574  | 2.867     | 0.33s                 | 0.00s                |

Seven bursts initially failed to measure at all. The cause: ffprobe -of csv=p=0 prints a trailing comma when a stream carries side data (1080,810,), so the width/height parsing broke on those companion files. Switching to JSON output brought them back.

The review videos were useless on the first pass: every clip forced to 480×270 landscape, labels silently dropped. My review note: “just one above the other, the ratio is not correct, what is this ?” The labels then broke a second time because a : in the text breaks FFmpeg’s filtergraph quoting. The tool I was using to check the fix needed its own fixes first.

What the models did, and didn’t

The March version was Claude. The September diagnosis and fix weren’t Claude or Codex: I ran them in opencode, first with GLM-5.3-flash, then with DeepSeek V4.1 flash when GLM started losing the plot (my exact words were “the previous work failed because the model hit its cognitive capacity I think”).

The good parts were real. GLM checked its own first theory against the code’s actual model and dropped it. It swept 25 saved plans from two years of renders. It printed the score for every offset before believing a minimum, and it found the shutter positions a second, independent way by matching stills to their own videos. That’s the grind I’d never have done by hand at 6 in the morning.

It also left a trail of self-corrections in a single session: “my min() was wrong”, “my stand-down premise was wrong”, “My sliver arithmetic was wrong”, “join_jump compares the after-join strip with itself”, a regex cleanup that deleted the script’s own entry point. Each of those was caught within the same session, mostly by the model itself. Fine.

The ones that mattered weren’t caught by the model. The midpoint fallback “strictly better than pure metadata”. The feature hidden behind a flag. The squashed review videos. And one I let through: the still-photo check and the correlation disagreed by 0.2 to 0.3 seconds on the race burst, and the issue comment called that “agrees in kind”. I should have pushed on that, and I still can’t fully explain it.

Same lesson as March, from the other direction. In March the model said “perfect” and the demo shipped. In September every automated check passed and the film jumped. Both times the thing that worked was watching the output.

What shipped

• Every multi-photo burst is stitched from measured offsets. The midpoint guess is gone from the render path.
• In the review report, measured joins came out at 0.00 s of rewind. The one rewind left (0.80 s) was a guessed join, which is why guessing is gone. The race burst went from a 4.42-second plan with two visible rewinds to 3.2 seconds of continuous footage.
• Two independent methods (motion and the March audio fingerprint) agree to within one probe frame on every join the matcher accepts.
• When one join can’t be measured, the burst shows as photos instead. Better a photograph than a stutter.
• The published docs demos were rebuilt with measured joins, which is the “after” side of the video at the top.

The matcher and its tests are in the repo: a textured image shifting one column per frame with a known nine-frame offset, flat and noise inputs that must be refused, and a real FFmpeg decode instead of preconstructed arrays. If you’ve got overlapping Live Photo bursts that break it, I’d love to see them.