The Search Bottleneck
As our internal RAG application grew to over 50 million embedded document chunks, our semantic search latency spiked past 200ms. Users were starting to notice the lag before the LLM even started generating text.
HNSW Hyperparameters
We were using an HNSW (Hierarchical Navigable Small World) index in our vector database. We had to dive deep into the hyperparameter tuning to fix the latency.
The two magic variables are ef_construction and ef_search.
// Example Qdrant Index Configuration
{
"hnsw_config": {
"m": 16, // Number of edges per node
"ef_construct": 200 // Higher = slower build time, better graph quality
},
"search_params": {
"hnsw_ef": 64 // Lower = faster query time, slightly lower recall
}
}By increasing ef_construction during index time (when we embed documents overnight), we built a highly optimized graph structure. By decreasing ef_search during query time (when the user is waiting), we sacrificed about 2% recall but slashed our latency down to 15ms. In production, 98% recall at 15ms is vastly superior to 100% recall at 200ms.