Sampling frames where the narrator points

A conservation nonprofit in Alaska had a recording of a Zoom call. A kelp farmer spent fifty minutes walking two of their staff through a state mapping site, showing which bays around Sitka could take a new farm and what would block a permit in each one. They wanted that call turned into something they could act on. Nobody was going to watch it again.

What they got

A briefing panel on the permit page of their operations dashboard. It holds three recommended sites, nine constraints, ten named places, eight map layers, three existing leases, two competing operators, and seven follow-ups, all pulled from the one call.

Every item carries the timestamp of the sentence it came from. Next to the timestamp is a play chip. Click it and a shared audio player seeks to that second and plays the farmer saying it. Every recommended site and every map layer also links to the frame of the screen-share from that moment, so the reader sees his cursor on the map, not my description of where it was.

The two sites he described clearly enough to draw got polygons. Those polygons are labeled as drawn from a frame, carry a confidence, and are clipped to water with the OpenStreetMap coastline so nothing crosses Baranof Island. The third site has no polygon, because he never pointed precisely enough to draw one, and the panel says so instead of guessing. Each site links into the permit drawing wizard with its rationale and polygon preloaded.

How the video became data

The farmer was already saying which seconds mattered. Every time he moved his cursor he said “here”, “right in here”, “this bay”, “south of”, “you can see”. The term is deictic language. So the pipeline is a regex over the transcript for those phrases.

Three rules made it clean. Skip the first three minutes, because a call opens with small talk over a title slide. Only count the farmer’s hits, since the staff also say “here” and they are not the ones driving the screen. Dedupe inside six seconds, because a person says “here” four times in a row about one place.

That pulled 119 frames out of 50 minutes. ffmpeg cut one JPEG per hit. I picked none by hand, and there was no moment in the briefing that lacked its frame.

Then one pass over the transcript and the frames wrote a JSON document: places, constraints, sites, layers, each with the timestamp it came from and the frame that shows it. The panel is a React component that renders that JSON. The one piece of engineering it needed was an audio player that could seek. Cloudflare Pages serves the file without range support, so the browser reports nothing seekable and drops any seek. The panel fetches the file once into a blob on the first click and points the player at that, which seeks fine.

The rule for the page was the same for every line: the reader can get from the claim to the moment that produced it in one click, and see what was on screen. A frame that no claim reaches is decoration. A claim that reaches no frame is hearsay.

Trying it on a narrator

I wanted to see whether the trick holds when the person pointing is a film narrator instead of a man with a cursor, so I ported the sampler to 300 public-domain films from the Prelinger collection, 66 hours of footage. The plate above is one of them.

The word list changed. Bare “this” was noise on the call, but a 1951 narrator says “this is the eye of a bee” and means it. The skip window shrank from three minutes to thirty seconds, because a film opens on a title card. A film with no narration yields nothing, which is correct. Across the first 85 narrated films the regex fired about 72 times an hour.

The first version of this plate showed a flaw. The sampler kept the start of the Whisper segment that matched, and the pointing word can land anywhere inside it. Measured across the 4,335 pointing segments in the 296 films that have a transcript, the median lag was 1.18 seconds and the 90th percentile 4.40, with 34 percent over two seconds and 13 percent over four. The narrator says “here’s the head of a worker” at 129.18 seconds and the sampler kept 122.76, which is a wide shot of several bees on comb with no worker’s head in close-up.

That was fixed on September 11. Whisper times every word and not only every segment, so the transcript rows keep the word times and the sampler takes the frame at the start of the word the regex matched. The plate above is redrawn from the fixed sampler, and three of its four blown-up frames are different pictures now. The first moved from 122.76 to 129.18, onto the close-up of a worker’s head with the probe on it. The second moved 1.10 seconds, one tile along. The third moved from 151.42 to 157.68, off a bee walking across the comb and onto the open cells the narrator is naming. The fourth moved 0.64 seconds and stays on the tile it was already on. The number of pointing hits in the film did not change, because the fix moves where a deictic frame sits and not how many there are.

The port feeds a clip search app with a recall eval behind it, comparing this policy against one frame a second. That page gets its own post when the numbers exist.

Second pass, 2026-09-13

The numbers exist now.

The transcription run finished the corpus it was partway through when this went up. All 300 films are transcribed. 241 of them have speech and 59 are silent or carry music only (four of those have no audio track at all), 365,425 words in total, about 2.8 hours of processing over four resumed runs on September 8.

That is the whole index, from the archive to the vectors. The app’s build log at clip-portal.eknuth.dev/log carries the same drawing. Its legend counts the frames as the first sampling run left them; the table below counts them after thirteen films were resampled on September 11, which moved uniform by 13 frames, scene cuts by 1 and guided by 4.

Every policy then ran over all 300 films, SigLIP embedded what each one kept, and a retrieval eval scored them through the same GET /api/search the pages call. Ground truth is 100 queries written by watching clips, never by looking at what the index returns. A row counts as a hit when the right film comes back at or above rank 10 and the second it returns is within 10 seconds of the second the query was written from. Thresholds are tuned on 80 of the rows and 20 are held back. The table is printed by tools/deictic-table.py out of the eval’s recorded run and pasted in rather than retyped, so rerunning the eval regenerates it.

The deictic and scene-cut rows are passes of the 100 ground-truth queries with no negatives and no controls, served by build 3e5fa41 and folded into the eval’s document afterwards; results.json’s rescored_from records which rows came from which pass, and the other four rows were served by 7f93177, with guided by 58b2d6c. The scene-cut row is the second of two passes. The daily search budget this project sets itself ran out partway through the first one and refused four queries, which the eval scores as misses, so the path ran again after the cap reset and answered all hundred. Both pass files are kept, because the gap between them is what a refused cell costs a score. Every figure below is scored before the score floor the live app applies, which is what makes this a comparison of samplers instead of a comparison of floors.

One frame a second wins this eval. Deictic on its own indexes 1 frame in 62 of uniform’s and returns about a quarter of its recall. Guided is deictic unioned with a scene-cut policy, and it costs a tenth of the index for 0.23 against uniform’s 0.38. On this run the pointing frames added nothing at rank ten over the cuts alone: both read 0.23, the same eighteen rows of eighty. On the two columns beside it the cuts alone read a little better, MRR 0.19 against 0.18 and a median seek error of 34.5 seconds against 35.4, though that median is over 68 films found where guided’s is over 69. The unfiltered row is what the app serves by default, and it tracks uniform, which is what should happen when uniform is nine tenths of the rows.

The seek error column is the part I had not thought about. Uniform misses the second by 10.8 seconds at the median and deictic by 84.4. A sparse index finds fewer films and lands further from the moment inside the ones it finds, because the nearest frame it kept can be a minute away from the thing the query described. The keyword path has the best seek error of the six at 2.9 seconds, since a transcript segment already carries the second it starts on, and it finds the fewest films of the six: it ranks the right film in the top ten for 32 of the 100 queries, where uniform does it for 82. A film has to say the thing out loud to be found that way.

What deictic still has is price. Its 3,881 frames are eleven minutes of embedding against uniform’s eleven and a half hours, and they are the only frames in the set chosen by what the narrator said about them. Spread over the 55.2 hours of the 241 narrated films, the regex fired about 70 times an hour across the whole corpus, which is close to the 72 an hour it was firing on the first 85 films.

That is one search, from the box to the frame. The same build log carries this drawing too.