Executive summary
Vector databases are the backbone of retrieval‑augmented generation (RAG). They store numeric embeddings produced by language models, index those embeddings for fast similarity search, and expose metadata columns that enable tenant isolation, time‑based filtering, and other business rules. The core trade‑off is between exact nearest‑neighbor search, which guarantees perfect recall but scales poorly, and approximate indexes, which reduce latency at the cost of occasionally missing the true nearest rows. Understanding how PostgreSQL with pgvector and Qdrant implement HNSW and IVFFlat indexes, how filters are applied after the index scan, and how to evaluate a pipeline on realistic queries is critical before trusting the retrieved evidence.
Introduction
Retrieval‑augmented generation (RAG) combines a large language model with a knowledge source that the model can query at inference time. The retrieval component turns a user question into an embedding vector, searches a vector store for the most similar chunks, and feeds those chunks back to the generator. The quality of the final answer depends on two independent factors: (1) the geometric similarity measured by the vector index and (2) the correctness of the surrounding metadata that determines whether a chunk is authorized and up‑to‑date. This article explains the technical fundamentals of vector storage, the indexing options available in open‑source projects such as pgvector and Qdrant, and a step‑by‑step approach to building, testing, and operating a high‑performance retrieval system.
Technical fundamentals
Embeddings and vectors
An embedding is a list of floating‑point numbers generated by a model that captures semantic meaning. In pgvector, a vector column can hold up to 16 000 dimensions, though most index types support fewer dimensions. The index type must be compatible with the vector width; HNSW and IVFFlat support up to 2 000 dimensions, while the half‑precision index supports up to 4 000.
Distance metrics
The index query chooses a distance operator. pgvector documents L2 (Euclidean), inner product, cosine, and L1 (Manhattan) distances. Changing the operator changes which rows are considered near because each metric measures similarity differently.
Vector tables also contain ordinary relational columns—tenant identifiers, document IDs, timestamps, access labels, etc. The index does not interpret these columns; they are used after the vector scan to enforce business rules. This separation is a source of confusion because a filtered result may contain fewer rows than the requested LIMIT, especially with approximate indexes.
Architecture and components
A typical RAG retrieval pipeline consists of the following components:
- Embedding service – Generates query and document embeddings.
- Vector store – Persists vectors and metadata. Example implementations include PostgreSQL with the pgvector extension and Qdrant.
- Index – Provides nearest‑neighbor search. Options are exact scan, HNSW graph, or IVFFlat list.
- Filter layer – Applies tenant, time, or policy filters after candidate vectors are returned.
- Grounding validator – Optionally re‑ranks or verifies retrieved chunks before passing them to the generator.
Detailed implementation approach
Schema design
CREATE TABLE documents (
id UUID PRIMARY KEY,
tenant_id UUID NOT NULL,
content TEXT NOT NULL,
embedding VECTOR(1536),
created_at TIMESTAMP WITH TIME ZONE,
tags TEXT[]
);
The embedding column stores the vector, while tenant_id, created_at, and tags are ordinary columns used for filtering.
Index creation
For exact search on a small table:
For larger tables, choose an approximate index. HNSW example:
CREATE INDEX ON documents USING hnsw (embedding) WITH (m = 16, ef_construction = 64);
IVFFlat example (requires data present before building):
CREATE INDEX ON documents USING ivfflat (embedding) WITH (lists = 1000, probes = 10);
The lists value follows the guidance of row count divided by 1 000 for tables up to a million rows, and the square root of the row count above that.
Query pattern
SELECT id, content
FROM documents
WHERE tenant_id = $1
ORDER BY embedding <-> $2
LIMIT 5;
When an approximate index is used, the filter on tenant_id is applied after the index returns candidate rows. If the filter discards too many candidates, pgvector from version 0.8.0 can perform iterative scans to fetch additional candidates until the LIMIT is satisfied.
Architecture and data‑flow
graph LR
A[User query] --> B[Embedding service]
B --> C[Query vector]
C --> D[Vector store (HNSW or IVFFlat)]
D --> E[Candidate rows]
E --> F[Metadata filter]
F --> G[Grounding validator]
G --> H[LLM generation]
H --> I[Response]
The diagram shows a query vector entering either an HNSW walk or an IVFFlat probe, followed by a metadata filter and optional grounding validation before generation.
Security and governance
Because the index does not enforce access control, the application must filter results by tenant or other policy attributes after the vector scan. Relying on the index alone can expose data from other tenants if a filter is omitted. Iterative scans mitigate the risk of under‑fetching when filters remove many candidates, but they do not replace proper authorization checks. Auditing should record the query vector, the filter parameters, and the final set of returned IDs to provide traceability.
Enterprise use cases
- Multi‑tenant knowledge bases – Each tenant’s documents are stored in the same table with a
tenant_id column. Retrieval must respect tenant isolation; the filter is applied after the index scan.
- Regulatory policy search – Documents carry timestamps and version tags. Queries must only consider records that are active at the time of the request, requiring a
created_at filter.
- Customer support chatbots – Recent chat logs are chunked and embedded. Fast approximate search (HNSW) provides low‑latency answers while iterative scans ensure enough authorized candidates are returned.
Benefits and limitations
| Index type | Build requirement | Default parameters | Recall‑vs‑latency tuning |
|---|
| Exact scan | None (sequential) | N/A | Perfect recall, latency grows with row count |
| HNSW | Can be created on empty table | m=16, ef_construction=64, ef_search=40 | Increase ef_search for higher recall, at cost of slower queries |
| IVFFlat | Requires data before build | lists ≈ rows/1000 (≤1M) or √rows (>1M), probes=1 | Increase probes to search more lists, improving recall but adding latency |
Limitations include the fact that approximate indexes may omit true nearest neighbors, especially when metadata filters discard many candidates. The index does not verify the factual correctness of retrieved text; downstream grounding or verification is required.
Operational considerations
- Index rebuilds – Changing vector dimensionality or index parameters often requires rebuilding the index. HNSW can be built incrementally, but IVFFlat needs a full rebuild when the number of lists changes.
- Monitoring – Track query latency, recall (measured against a labeled test set), and filter discard rates. High discard rates may indicate that the index is not returning enough candidates for the filter to satisfy the LIMIT.
- Backup and restore – Vector columns are stored like any other PostgreSQL column, so standard logical or physical backups capture the embeddings.
- Version upgrades – pgvector 0.8.0 introduced iterative scans; upgrading can improve recall for filtered queries without changing application code.
Best practices
- Start with exact search on a development dataset to establish a baseline of true nearest neighbors.
- Profile latency as the table grows; switch to HNSW or IVFFlat when latency exceeds service‑level targets.
- Tune distance metric to match the semantic behavior of the embedding model (cosine for normalized vectors, inner product for similarity‑oriented models).
- Apply filters after the index and verify that iterative scans are enabled if filter discard rates are high.
- Validate retrieval with a labeled question‑answer set that respects tenant and temporal constraints; measure recall on the authorized subset, not on the whole table.
- Document index parameters (m, ef_construction, ef_search, probes) and version them alongside schema migrations.
Analysis
The sources clarify that vector indexes provide geometric similarity but do not guarantee the truthfulness of the retrieved text. Exact search offers perfect recall of geometric neighbors, while HNSW and IVFFlat trade recall for latency. The documentation emphasizes that filters are applied after the index scan, meaning that a filtered query can return fewer rows than requested, and that iterative scans (pgvector ≥ 0.8.0) can mitigate this by re‑probing the index. Qdrant adds the concept of a payload index to make filter‑aware graphs, but it still requires rebuilding the graph when payload indexes change. Therefore, system designers must treat the index as a fast geometry engine and place responsibility for policy enforcement, grounding, and factual verification in downstream components.
What the sources do not settle
- Absolute recall percentages for HNSW or IVFFlat on real‑world corpora. The documentation provides only directional guidance (higher
ef_search or more probes improve recall) without quantitative benchmarks.
- Performance comparison between pgvector and Qdrant under identical workloads. Each project describes its own trade‑offs, but no side‑by‑side latency or resource usage numbers are given.
- Optimal default parameters for specific workload sizes. The suggested defaults (e.g.,
m=16, ef_search=40) are starting points; the best values depend on vector dimensionality, hardware, and query patterns.
- Impact of embedding model drift on retrieval quality. While the article notes that outdated embeddings can return stale chunks, the sources do not quantify how often this occurs or how to detect it automatically.
Conclusion
Vector databases such as PostgreSQL with pgvector and Qdrant provide the essential building blocks for retrieval‑augmented generation. By storing embeddings, configuring appropriate indexes, and applying metadata filters after the vector scan, developers can achieve low‑latency, high‑recall retrieval that respects tenant isolation and other policies. Exact search guarantees geometric recall but does not scale; HNSW and IVFFlat offer configurable trade‑offs. Evaluating the pipeline with real questions, measuring recall on the authorized subset, and employing iterative scans where needed are necessary steps before trusting the retrieved evidence for generation.
References
pgvector documentation
Qdrant indexing concepts
These published CloudExchangers articles are further desk reading. They are not sources for the technical claims above.