Why AI search feels slow even with good models
AI search often feels slow because the “model” is only one step in a longer chain. A typical request has to turn the query into an embedding, hit storage for candidates, apply filters (tenant, ACLs, recency), rerank results, and sometimes generate an answer. Each step is small, but the round trips add up.
As the corpus grows, retrieval stops being a simple lookup and becomes a distributed fan-out: more shards to query, more candidates to merge, more cache misses, and more tail-latency from the slowest node. Even strong models can’t compensate for a pipeline that spends most of its time waiting on I/O.
Making it fast usually means paying something: more memory for caches, more replicas to cut tail latency, or more precomputation for indexes. Those costs rise quickly once you move from “demo-scale” to real traffic and real datasets.
What “large-scale databases” actually change in the pipeline
When people say “use a large-scale database,” they usually mean changing where time is spent between retrieval and merge. Instead of every request scattering across many machines and stitching partial answers in the app, the database takes on more of the work: pruning candidates early, running filters close to the data, and returning a smaller, cleaner set to rerank.
That shows up as concrete features: partitioning/sharding so queries touch fewer nodes, replicas to reduce tail latency, and indexes that avoid scanning (keyword, vector, and metadata/ACL). Query planners matter too, because a good plan applies cheap filters first and only runs expensive similarity search on what survives.
More indexes mean more write amplification, more memory pressure, and longer rebuilds when schemas or embedding versions change.
Indexing choices that turn big corpora into fast retrieval
In practice, “fast retrieval” comes down to how aggressively your indexes let you avoid touching most of the corpus. For text, a well-built inverted index can narrow millions of documents to thousands in a few milliseconds, especially when you index the same fields you later filter on (tenant, language, doc type). For vectors, approximate nearest neighbor (ANN) indexes work the same way: they trade a small amount of recall for predictable latency by probing a compact structure instead of scanning every embedding.
The key choice is what you index together. If metadata filters are applied after vector search, you waste work scoring candidates you’ll discard. If filters can be pushed into the index path—via pre-partitioning, per-tenant indexes, or “filtered ANN” support—you cut both CPU and network. You also need a plan for change: every extra index increases write amplification, and rebuilding after an embedding model upgrade can be days of backfill unless you support dual-read or versioned indexes.
Hybrid search: blending keywords, vectors, and filters without stalling
A common pattern in production is a query that mixes “exact” intent with “fuzzy” intent: a user types a product name, adds “error 504,” and expects results only from their workspace and the last 30 days. If you run vector search first, you may score thousands of semantically similar chunks only to throw most away on tenant, ACL, and time filters. If you run keyword first, you may miss relevant paraphrases unless you keep recall high.
Hybrid search stays fast when it’s treated as candidate generation plus controlled reranking. The cheap signals (tenant partition, ACL bitmap, doc type, time range, keywords) should shrink the search space before ANN probing or at least before full scoring. Many teams use a two-lane approach: retrieve top-K from an inverted index and top-K from ANN, union them, then rerank with a heavier model.
Pick K too low and recall drops; too high and latency spikes, especially at p95 when one shard returns a large candidate set.
Speed at scale comes from layout, caching, and locality
Once you’ve controlled candidate counts, latency starts to hinge on where the bytes live. If your vectors, postings, and metadata for a document are stored far apart—different nodes, different disks, different services—every query becomes a small distributed join. Co-locating “things you fetch together” (vector + doc id + filter fields, or precomputed ACL/tenant bitmaps) reduces random reads and cross-shard chatter, which is often what drives p95.
Caching is the other lever, but it only works when you cache the right layer. Query-result caches help for repeated queries, but many products see long-tail queries where that cache misses. More reliable wins come from caching hot index pages, filter bitsets, and “top candidates per partition,” so the first hop is memory instead of disk. Replicas can also act like a cache by increasing the chance a request lands on a warm node.
Larger caches mean more RAM, and high update rates can invalidate cached structures quickly. If your working set doesn’t fit in memory, layout and locality become the difference between steady throughput and unpredictable tail latency.
Freshness and updates: staying fast while the data changes
Teams usually notice freshness problems first as “why can’t I find what I just uploaded?” and “why did latency jump after we turned on near-real-time indexing?” Updates break the assumptions that make retrieval fast: caches go stale, segments need merging, and ANN structures that were compact yesterday start to fragment or fall back to slower paths when they’re constantly being patched.
Keeping p95 steady often means separating “write-fast” from “read-fast.” New documents land in a small, append-only buffer (or a lightweight index) that’s cheap to update, while the main indexes are rebuilt or merged in the background. Queries then hit both: the fresh layer for recency, the optimized layer for scale, and a dedupe step in between. This adds a little merge cost per query, but avoids full reindexing on every write.
The practical constraint is operational: background rebuilds consume CPU and I/O, and dual-read or versioned indexes require extra storage and careful cutovers during embedding upgrades.
Choosing a stack: search engine, vector DB, or “one database”
The decision usually shows up as a familiar tension: do you want one system to answer “find relevant things” end-to-end, or do you want a few specialized systems that each do one part fast? A classic search engine (Elasticsearch/OpenSearch/Solr) is hard to beat for keyword + metadata filtering, and it tends to have mature tooling for shards, replicas, and query profiling. If most of your latency comes from filters, ACLs, and “must include these terms,” starting there and adding vectors incrementally can be the lowest-risk path.
A vector database is attractive when the primary bottleneck is nearest-neighbor search at scale and you need predictable ANN performance, but many teams still end up rebuilding missing pieces around it: robust faceting, complex boolean filters, and operational guardrails for freshness. “One database” options (systems that claim keywords, vectors, and filters together) can reduce network hops and simplify consistency, but you’re betting on one query planner and one index implementation being good at all three.
The practical test is workload-shaped: take a week of real queries, replay them, and compare p95 with your full filter set and update rate. The wrong choice often looks fine on recall benchmarks and falls apart on tail latency once shards, caches, and rebuilds enter the picture.
Putting it together: a practical blueprint for faster AI search
A practical blueprint is to treat retrieval as a budgeted pipeline: (1) pre-partition by tenant (or another hard boundary), (2) push ACL and time filters into the earliest step you can, (3) generate candidates from two cheap lanes (inverted index top-K and ANN top-K), then (4) rerank a bounded union with your heavier model. Keep vectors and filter fields co-located, and cache index pages plus hot filter bitsets rather than full results.
For freshness, use a small “recent” layer plus an optimized main index, and merge at query time with dedupe. Expect costs: extra RAM for caches, write amplification from multiple indexes, and background rebuild I/O during embedding upgrades.
