Vector database architecture and design patterns.
Choosing a vector database is the easy part. Picking the right index, chunk size, and scaling pattern is what determines whether your RAG system meets recall and latency targets. Here's the design space.
When NOT to use a vector database
- • Exact ID lookup—use a key/value store.
- • Fuzzy search on rare entities (names, SKUs)—use BM25 or trigram.
- • Real-time leaderboards or rankings—use a sorted set.
- • Strict transactional consistency on writes—use a relational DB.
When to use one
- • Natural-language queries over unstructured text.
- • Semantic similarity between items (recommendations, related-content).
- • Deduplication of near-identical records.
- • Clustering and topic discovery.
Index types
- • HNSW (hierarchical navigable small world): high recall, slower updates—what CogniWeave uses by default.
- • IVF-PQ: lower memory and higher throughput, at some recall cost—good for very large corpora.
- • ScaNN / Annoy: alternative trade-offs; ScaNN excels at high-recall + high-throughput on Google hardware.
Chunking strategy
- • Small chunks (~128 tokens): higher recall, finer granularity, more results to rerank.
- • Large chunks (~512 tokens): more context per result, fewer fragments, faster end-to-end.
- • Hybrid: retrieve with small chunks, then expand to parent chunks for the LLM context window.
Scaling patterns
- • Single node: up to ~50M vectors fits comfortably on commodity hardware.
- • Sharding: hash the query namespace to route to a shard; merge top-k across shards.
- • Replication: leader–follower for read scaling and zone redundancy.
- • Tiered storage: hot vectors in RAM, warm in NVMe, cold paged from object storage.
Common questions
How do I update or delete vectors?
HNSW supports inserts and soft deletes in-place; periodic compaction reclaims space. CogniWeave handles compaction automatically and exposes upsert/delete by source document or chunk ID.
How do I bulk-ingest a large corpus?
Stream documents through the ingestion API; embeddings and index writes are batched. Expect roughly 1k–5k chunks/second per worker depending on chunk size and embedding throughput.
Can I filter results by metadata?
Yes. Pre-filtering narrows candidates before ANN; post-filtering applies after retrieval. CogniWeave supports both and picks the better strategy based on selectivity.
Which distance metric should I use?
Cosine for normalized embeddings (the default for most transformer models, including CogniWeave's managed embeddings). Inner-product if your embeddings are unnormalized. L2 for image/audio embeddings where magnitude carries signal.