Two stores, one eval

Type “a small airship inside a hangar with men working under the nose” into the search box at clip-portal.eknuth.dev. The first result is a 1931 film at 1 minute 38 seconds, with the player already seeked there. The airship is on screen two seconds later. Nobody tagged that second. When the index was built, frames were pulled out of the 66.4 hours of footage, 264,626 of them. An image model turned each frame into a vector. A vector is a fixed-length list of numbers, arranged so that two pictures of the same kind of thing get lists that sit close together. The words in the search box get a list of the same shape. A search is a hunt for the frames whose lists are nearest to it.

Those 264,626 vectors lived in one place all project: pgvector, with an HNSW graph over the column. pgvector is the extension that lets Postgres keep a vector in a column and measure the distance between two of them. HNSW is an index that links each vector to its near neighbours, so a search can walk from one to the next instead of measuring the distance to all 264,626. Last week I wrote the same vectors a second time, as a Lance dataset in an object store. Lance is a file format built for vectors. An object store is a bucket of files served over HTTP the way S3 is. I served search out of that copy through the same route the page calls, and ran the same hundred-query eval against both. Then I added a third column that nobody in this project built: a commercial video embedding model, over the 14 percent of the corpus its free tier would hold.

How the eval scores a search

The eval is 100 queries written by watching clips, never by looking at what the index returns. Each one names a film and a second inside it. A path is which index answers the query. The keyword path searches the transcripts. The frames paths search the frame vectors: all of them (none), the ones a one-frame-a-second sampler kept (uniform), or the ones kept by a policy that follows what the narrator says (guided).

Three columns score what comes back. Recall@10 counts a query as a hit when the right film is in the top ten results and the returned second is within ten seconds of the one the query was written from. “Tuned” means it is scored over the rows the score floor was tuned on, with a held-back set kept apart. MRR is mean reciprocal rank. A right film at rank one is worth a full point, at rank two half a point, at rank three a third, averaged over all hundred. Seek error is how far the returned second lands from the one the query came from, in seconds. It is taken at the median over only the queries where the film was found at all, and the n in the cell is how many that was.

What the two stores return:

The other three columns are what a query costs. The route times its own nearest-neighbour step, the part of a request that finds the closest vectors. Turning the words into a vector and reading the film rows are timed apart from it. p50 is the median. p95 is the time that 95 requests in 100 come in under. A cold request is one sent when whatever answers it had been asleep or freshly started, so it pays the cost of waking them. The worst cold request in a pass gets its own column.

What a query over them costs:

Both tables are printed out of the eval’s own results file by a script, so a rerun regenerates them. The keyword row has one store because the transcript index has no second copy. Two rows read “not recorded” in the cold column because the pgvector passes predate the dispatch stamp. That is the timestamp the runner now writes when it sends a request, and it is how the runner tells the first request of a pass from the rest. On those passes the first request is counted warm, and the only cold cell they can carry is a retry.

Neither store won. Sampling decides how many vectors exist: uniform keeps 239,111 and the narrator-guided policy 24,734. The store decides what a query over them costs, in time and in money. Two decisions, measured apart. That is why they are two tables.

What matched

The guided rows are identical to the last digit on every cell the eval prints: recall, MRR, seek error, the negatives rejected and the controls cleared. Negatives are twenty queries for things the corpus does not hold. A path gets them right by returning nothing above its score floor. Controls are seven ordinary queries the corpus does answer. A path gets those right by letting them through. On the guided path the two stores returned the same top result on 98 of 100 queries, and on the two unfiltered paths 95 and 94. On every path the two stores land within two rows of a hundred of each other on recall@10.

The two queries where the unfiltered paths part both went Lance’s way. One asks for a giant open steel globe sculpture against the sky. The other asks for a line of narration about a Chicago dialling code. On both, the HNSW walk returned a weaker frame of the right film and pushed it down the list, to rank twelve on uniform. The Lance index re-reads the exact vectors of its candidates before ranking them. It returned that film’s best frame and ranked it first and second.

What a query cost in each

A Lance query costs about the same on every path. A pgvector query costs whatever its index is. Lance’s median nearest-neighbour step spans 136 ms across the three paths, 664 to 800. pgvector’s spans 837, from 98 to 935.

The Lance index sorts the vectors into 256 partitions by rough similarity. A query probes the 64 partitions nearest to it, then re-reads the exact vectors of the 400 best candidates to rank whatever the policy filter keeps. That is 38 requests for byte ranges of the dataset files per warm query, about 11 ms each at the 438 ms median that tower.lance_query reports. Lance queries run inside the text tower, the container that turns the query words into a vector. That span is the container’s own timing of the step. Postgres reads whatever index the planner picks. The guided path got an index of its own, 193 MB against the full 2 GB, so it answers in 98 ms. Lance wins one path here, uniform, by 135 ms at the median. It loses guided by eight times.

Where each store goes cold

tower.lance_query read 11,561 ms on the container’s first Lance query. That container had started from a fresh image with an empty index cache. The three queries after it read 4,475, 3,132 and 2,721 ms, as partitions the first query had not probed came in. Opening the dataset cost 302 ms once, for the container’s life.

Postgres pays the same kind of cost in a different place. It is a hosted database whose compute switches off after five idle minutes and back on at the next query. On a freshly resumed one, six queries nobody had asked before, q010 through q060, cost 4,320, 6,137, 5,091, 4,584, 5,550 and 4,341 ms, with no decay. The eval’s own cold column tops out at 2,525 ms because the container had answered those first four Lance queries before the pass started. The runner waits for the tower to come up. That wait polls a health endpoint and never touches the dataset.

A column somebody else’s model built

Everything above runs on one image model, SigLIP. It maps a picture and a piece of text into the same vector space, so a sentence can be compared to a frame. It ran over frames this project chose, on a Mac. That is a closed loop. The eval measures a pipeline against ground truth written by the same person who built the pipeline. It can say which sampler is better without saying whether the whole approach is any good. So I put a commercial video embedding model in next to it: Marengo 3.0, through TwelveLabs’ free tier.

The vendor sells a managed Search API and an Embed API. The column uses the Embed API, which matters for what the eval can claim. The Embed API sells the vectors and lets them go. They land in this project’s own Postgres, and path=marengo is a plain HNSW query of the same shape as frame_search, the SQL function behind every frames search. A column that proxied out to a rented index on every query would have measured somebody else’s whole stack through a network hop. This one measures a retrieval path this eval serves.

The free tier caps indexing at ten hours, against a corpus of 66.4. The films were chosen the way the ground truth was: by the ground truth, never by the index. I took the archive ids named in queries.jsonl, in byte order, and added films until the budget ran out. 42 films went in, 5,456 segments of about six seconds each, and the vendor’s meter closed at 9 hours 51 minutes. One film was refused for being 320 by 240, under the vendor’s resolution floor. The last four were refused when the allowance ran out.

So the vendor holds the film for 42 of the hundred queries. It was never given a chance at the other 58. Those 58 are printed as not covered rather than as misses. Every path it is read against is rescored over the same 42, because a rate over a hundred rows and a rate over these 42 are different measurements.

It wins. The first thing to say about that is that the two sides searched different haystacks. Marengo searched 42 films. Every SigLIP path searched 300. So the vendor was picking out of a haystack seven times smaller. A smaller haystack flatters the film-finding columns most, and film in top 10 is the column the vendor leads by the widest margin.

Two things survive that objection.

The first is where the gap comes from. Of the 42 covered queries, 21 are visual and 12 are vague. On all 33 of those, both Marengo and uniform find the right film every single time. The entire film-finding gap is the nine spoken queries, the ones that quote a line of narration. Marengo lands the film on eight of nine. A frame model lands it on three. Marengo reads the audio track and SigLIP looks at pictures. The gap on spoken queries comes from a sense the frame index lacks, and a smaller haystack cannot produce it. The keyword leg, which reads transcripts, lands the film on all nine of them and on almost nothing else.

The second is the seek. Once a path has the right film, where inside it the result lands is decided within that one film, however many others were in the pool. Marengo’s median miss is 6.5 seconds. Uniform’s, on the same 42 queries, is 8.5. Of the films each path finds, Marengo puts the viewer within ten seconds of the moment 58 times in a hundred, and uniform 50. That difference survives the haystack objection. Keyword’s 3.0 seconds is the sharpest number in the column and the least useful. It is a median over the 13 films the transcript index found at all.

A Marengo query is also the fastest thing on the site: 19 ms at the median and 41 at p95. That is what a nearest-neighbour walk over 5,456 rows should cost. The number is small for the same reason the recall is flattering. Its score floor, the similarity below which a result is dropped, came out of the same twenty negatives as the others, at 0.0749. It turns away 5 of the 16 tuned negatives at no cost to a hit. The frame paths turn away 11, and the keyword leg all 16.

What a result is

Every row in both stores carries the film, the second inside it, and the policy that kept the frame. So a result is a place in a film. The page seeks the player to it and plays from there. The vector is an index entry. The thing anyone would license is the film.

A vendor row is a stretch of time rather than an instant. Marengo cuts a video into its own six-second clips and returns one vector per clip. The card says segment 4:24 to 4:30 and the player seeks to the start. It says so rather than rounding the stretch down to a frame stamp this project did not choose.

What a month costs

The month I planned for is about $11 against a $25 ceiling. $5 of that is the Workers subscription, the flat charge for the plan the app runs on, owed whatever happens. $4.60 is database: $3.18 compute and $1.40 storage for four gigabytes of frame vectors and their graph. $0.23 is object storage for 25.1 GB of transcoded film. Under $0.50 is container. What the billing page read on the 11th was $1.41 to date, $1.30 of it compute over 12.22 CU-hours at $0.106 an hour and $0.10 of it storage. A CU-hour is the database’s meter, one unit of compute switched on for one hour. So the database line is measured, and the month it is heading for is the estimate.

The container line is modeled either way. The Worker records each container start, and the checker charges one as eleven minutes of a standard-2, the container size it runs on. Reading the real meter needs an account-scoped token this project has not set. The Lance dataset adds about two cents, 1.529 GB at $0.015 per GB-month. Its reads are billed per million operations with no egress charge, so they round to nothing here. Making it the default would take most of the $1.40 of storage and none of the compute. The film metadata, the catalog counts and the keyword path still read Postgres on every search.

The vendor column cost $0. The free tier is also the reason it stops at 42 films. TwelveLabs’ own indexing rate is $2.50 an hour, so the 66.4-hour corpus would be $166 to embed once. That is more than six times the monthly ceiling this whole project runs under, and it is why the column is a slice and not a default. The comparison the table is really making is between somebody else’s model at $166 a corpus and two cents a month to keep a copy of my own vectors on object storage. On the slice that was measured, the expensive one gives the better answer. This eval says nothing about whether it still would over 300 films.

What the one store had already taught

Before the second store there was one. Four of the things it taught are store findings rather than sampling findings. The budget that measured itself has the measurements behind each. This is the short form.

The statement timeout, Postgres’s cap on how long one query may run, was never in effect. Passed to the database client as a startup setting, it did nothing through the connection pooler. A query given 50 ms ran for 413, twelve times out of twelve. It is set inside a transaction now.

The first search on a cold database failed on its own. The hosted database suspends after five idle minutes, and the first query after that pays for waking it. Raising the timeout to cover that moved the cancellations to the new ceiling.

Postgres caches a query plan per session. From the sixth run it may swap in a generic plan that ignores the actual parameter values. That generic plan left the vector index, and one execution cost 41,367 ms. Forcing a custom plan holds every execution at 47 to 53 ms.

The guided path was being answered by a scan of the whole table instead of a walk of its index. The planner priced the scan on the small main table and counted the vectors it had to fetch alongside as free. Three of the five queries the app can make were doing that. With enable_seqscan = off scoped to the function, the guided query went from 497 to 874 ms down to 61 to 67.

How it was measured

The eval calls GET /api/search on the live site, the same function the pages call. There is no separate eval mode. The store is a query parameter that moves which index answers, and so is the vendor path. The grouping of frames into films, the keyword leg, the merge of the two on the hybrid path, the floor and the render are the same code on all of them. Each frames path sent 127 requests per store: 100 hand-written ground-truth rows plus 20 negatives and 7 controls. The keyword path sent 120, because the controls run on the frames paths only. The vendor pass sent the same 127, and its rates are taken over the 42 whose film it holds. The Cloudflare zone in front of the site rate-limits the route to 5 requests per 10 seconds, so dispatches pace 2.6 seconds apart. That sets a pass’s length. The three Lance passes took five and a half minutes each.

Every percentile above is the route’s own Server-Timing over the ground-truth requests, the only ones it timed. Server-Timing is a response header where a server reports how long each stage of a request took, so the number is the route’s own clock. A cold cell is the first request a pass dispatched, plus any cell that needed more than one try. Everything else is warm. The tracing backend, SigNoz, records each stage of a request as a span with a start and an end, and its spans agree. I lead with the route’s own numbers because of the tail sampler. The tail sampler decides after a request finishes whether to keep its spans. It exports every request over two seconds and one in five of the rest, so it thins away the fast searches. On guided it kept 121 spans of 127 and read 1,043 ms at p95, where the route’s own number is 1,141.

What this does not show

The catalog is 300 films, 66.4 hours, 264,626 frames. This measures that size and only that size. The sampling policy’s claim is that it lets a catalog grow without the index growing with it. A bigger catalog is what would test that.

There is one container instance with one index cache. The 11.6 second first query is paid once per instance, so a fleet pays it per instance. The warm numbers assume a cache that a busier deployment would be evicting from.

There is one query mix and it is text, which is what the refine step is tuned for: a query vector landing between the image vectors the partitions were trained on. A workload of stored-vector lookups would measure how well an index finds a row it already holds. That is an easier question.

The vendor column is 42 films and 42 queries. It says what Marengo 3.0 did on the slice a free tier paid for, against paths searching seven times as much. It separates what the audio gave the vendor from what the smaller haystack gave it only where the style breakdown lets it. It measures nothing about the whole corpus. With one vendor in the table there is no way to tell whether Marengo is unusually good or whether any video model that hears the audio would do the same.

The app is live at clip-portal.eknuth.dev. The build log, one entry per landed issue, is at clip-portal.eknuth.dev/log. The repository opens when the piece ships.