Vector search internals: HNSW, quantization, and filtered search
A vector database is an ANN index with knobs. Understanding HNSW's parameters, quantization, distance metrics, and filtered search is what lets you trade recall, latency, and memory deliberately at scale.
The vector-databases post said a vector DB runs approximate nearest-neighbour search over HNSW or IVF indexes. This one opens that box, because at scale the defaults are rarely right, and the only way to trade recall against latency and memory on purpose is to understand the knobs.
HNSW: the graph and its three knobs
HNSW builds a layered graph where each vector links to its nearest neighbours; a search walks the graph greedily from the top layer down. Three parameters set the trade-offs:
- m — the number of connections per node. Higher m means better recall and more memory (and slower builds).
- ef_construction — the size of the candidate list while building the graph. Higher means a better-quality graph but a slower, costlier build.
- ef_search — the size of the candidate list at query time. Higher means better recall but slower queries. This is the knob you tune per-query to move along the recall/latency curve.
IVF and quantization: trading accuracy for memory
IVF (inverted file) clusters vectors into cells and, at query time, searches only the nearest nprobe cells — fast and approximate, with nprobe as its recall/latency dial. Quantization compresses the vectors themselves: product quantization (PQ) splits a vector into sub-vectors and codes each against a small codebook; scalar quantization (SQ) shrinks each dimension to fewer bits; binary quantization goes all the way to one bit per dimension. All trade a little accuracy for large memory and speed wins — which is what makes 100M+ vector indexes affordable at all.
Distance metrics, and why normalization matters
Cosine (angle between vectors), dot product, and L2 (Euclidean distance) measure similarity differently. For normalised vectors, cosine and dot product rank identically; many embedding models are trained to be used with normalised vectors and cosine similarity. The rule is simple: use the metric your embedding model was trained for, and normalise consistently on both index and query — a mismatch quietly wrecks recall.
Filtered vector search: pre vs post
You usually want 'nearest vectors where tenant = X', and how the database applies that filter has an accuracy cost. Pre-filtering restricts the set first and searches only within it — accurate, but it can defeat the ANN index, which assumes the full graph. Post-filtering runs the ANN search and then drops non-matches — fast, but it can return too few results after the filter bites. Good vector databases integrate the two; know which yours does, because it affects both recall and, crucially, your tenant-isolation guarantee (a filter you rely on for isolation must not be best-effort).
The trade-off at scale
At 100M+ vectors you can't maximise recall, minimise latency, and minimise memory at once — pick the two that matter and spend the third. Then measure: benchmark recall@k against a brute-force baseline on your own data and query distribution, because the right m, ef_search, and quantization for your workload are an empirical question, not a default.
A vector database isn't a magic box — it's a graph with three dials and a compression scheme. Turn them on purpose, and measure recall on your own data, or you're just trusting someone else's defaults.