Self-Hosted Vector Database for AI: Milvus, Qdrant, and Weaviate Compared

Quick Summary (TL;DR)

• Milvus → Best for billion-scale datasets; microservices architecture, GPU support, but the heaviest to operate.
• Qdrant → Best for low-latency, high-throughput RAG apps; single binary, Rust-based, easiest to run.
• Weaviate → Best for hybrid search (vector + keyword) with a built-in GraphQL API and modular ML integrations.
• All three scale horizontally, support Docker Compose for local dev, and offer production-grade Helm charts.
• Pick based on scale, latency, and feature needs — not on raw benchmark numbers alone.

Vector databases have become the backbone of modern AI applications — from retrieval-augmented generation (RAG) pipelines to recommendation engines and semantic search. Managed services are convenient, but as your data grows and compliance requirements tighten, self-hosting becomes a strategic decision rather than an afterthought. In this guide, we’ll compare three of the most popular open-source options — Milvus, Qdrant, and Weaviate — and walk through deployment, tuning, and scaling.

Why Self-Host a Vector Database?

Managed vector databases charge per-vector or per-query, and costs can spiral once you cross millions of embeddings. Self-hosting gives you:

• Data sovereignty — embeddings often encode sensitive information; keeping them on-premises or in your VPC satisfies GDPR, HIPAA, and internal governance.
• Predictable costs — you pay for hardware, not per-query. At scale, this is often 5–10x cheaper.
• Customization — tune indexes, quantization, and replication to your workload instead of accepting a vendor’s defaults.
• No vendor lock-in — the data and schema remain portable.

The trade-off is operational responsibility: you own upgrades, monitoring, and capacity planning.

Key Considerations for Self-Hosted Vector Databases

Performance and Scalability

Raw QPS and recall numbers depend heavily on index type (HNSW, IVF, DiskANN), dimensionality, and hardware. What matters more is how each system behaves as you grow from 1M to 1B vectors.

Ease of Deployment and Management

Qdrant ships as a single binary with minimal dependencies. Weaviate is a single Go binary plus optional modules. Milvus is a distributed system with multiple components (etcd, MinIO, Pulsar/Kafka) — powerful, but heavier.

Feature Set and Ecosystem

Look for filtering (metadata pre/post-filtering), hybrid search, multi-tenancy, quantization, and client SDK coverage. All three support Python, Go, Java, and JavaScript clients.

Milvus: Overview and Architecture

Milvus uses a disaggregated, microservices architecture with separate layers: access, coordinator, worker, and storage. This design separates compute from storage, enabling independent scaling.

flowchart TD
    Client[Client SDK] --> Proxy[Proxy / Access Layer]
    Proxy --> Coord[Coordinators]
    Coord --> QueryNodes[Query Nodes]
    Coord --> DataNodes[Data Nodes]
    Coord --> IndexNodes[Index Nodes]
    QueryNodes --> Storage[(Object Storage / S3)]
    DataNodes --> Storage
    IndexNodes --> Storage
    Coord --> Meta[(etcd Metadata)]
    Proxy --> MsgQ[(Message Queue)]

Deployment Options

• Milvus Lite — embedded Python library for local prototyping.
• Standalone — single-node Docker Compose for dev/staging.
• Cluster — Helm chart or Milvus Operator on Kubernetes for production.

Performance Benchmarks

Milvus consistently leads on billion-scale workloads. Its DiskANN and GPU_CAGRA indexes allow serving enormous datasets from disk with GPU-accelerated search, achieving high recall at low latency.

Scaling Strategies

Scale query nodes for read throughput, data nodes for ingestion, and index nodes for build parallelism. Object storage (S3/MinIO) scales independently.

Qdrant: Overview and Architecture

Qdrant is written in Rust and built around a single-binary design with a focus on low latency and rich payload filtering.

flowchart LR
    Client[Client] --> API[REST / gRPC API]
    API --> Collections[Collections]
    Collections --> Shard1[Shard 1 - HNSW]
    Collections --> Shard2[Shard 2 - HNSW]
    Shard1 --> Disk[(NVMe Storage)]
    Shard2 --> Disk
    API --> Raft[Raft Consensus]

Deployment Options

• Docker — single container, quickest start.
• Docker Compose — with optional distributed mode.
• Kubernetes — official Helm chart with sharding and replication.

Performance Benchmarks

Qdrant excels at low-latency queries on moderate datasets (10M–100M vectors). Its filtering engine uses filterable HNSW indexes, avoiding the recall loss that plagues post-filtering approaches.

Scaling Strategies

Qdrant scales via sharding (splitting a collection across nodes) and replication (Raft-based consensus for high availability). Resharding is supported for live clusters.

Weaviate: Overview and Architecture

Weaviate is a Go-based vector database with a GraphQL-first API and a modular architecture for vectorizers and rerankers.

Deployment Options

• Docker Compose — the recommended starting point.
• Kubernetes — Helm chart with multi-node clustering.
• Weaviate Cloud — managed alternative, if you later change your mind.

Performance Benchmarks

Weaviate performs strongly on hybrid search workloads, blending BM25 keyword scores with vector similarity. Standalone vector search is competitive but typically behind Qdrant on pure latency.

Scaling Strategies

Weaviate uses a shared-nothing cluster: add nodes, and shards rebalance. Replication factor controls redundancy.

Comparison Table: Milvus vs. Qdrant vs. Weaviate

Feature	Milvus	Qdrant	Weaviate
Language	Go / C++	Rust	Go
Architecture	Microservices	Single binary / sharded	Shared-nothing cluster
Best Scale	Billions	10M–100M+	Millions–100M
Hybrid Search	Limited (via plugins)	Sparse vectors	Native BM25 + vector
Filtering	Strong	Excellent (filterable HNSW)	Strong
GPU Support	Yes	No	Yes (via modules)
Multi-tenancy	Yes	Yes (payload-based)	Yes (native)
Ease of Ops	Complex	Easy	Moderate
API	gRPC / REST	REST / gRPC	GraphQL / REST
License	Apache 2.0	Apache 2.0	BSD-3

Step-by-Step Deployment Guide

Prerequisites

• Docker Engine 24+ and Docker Compose v2
• 16 GB RAM minimum (32 GB+ recommended)
• NVMe SSD for production
• curl and jq for testing

Deploying Milvus with Docker Compose

wget https://github.com/milvus-io/milvus/releases/download/v2.4.0/milvus-standalone-docker-compose.yml -O docker-compose.yml
docker compose up -d

This spins up Milvus, etcd, and MinIO. Verify:

curl http://localhost:9091/healthz

Deploying Qdrant with Docker Compose

services:
  qdrant:
    image: qdrant/qdrant:latest
    ports:
      - "6333:6333"
      - "6334:6334"
    volumes:
      - ./qdrant_storage:/qdrant/storage

docker compose up -d
curl http://localhost:6333/collections

Deploying Weaviate with Docker Compose

services:
  weaviate:
    image: cr.weaviate.io/semitechnologies/weaviate:1.25.0
    ports:
      - "8080:8080"
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "true"
      PERSISTENCE_DATA_PATH: "/var/lib/weaviate"
    volumes:
      - ./weaviate_data:/var/lib/weaviate

docker compose up -d
curl http://localhost:8080/v1/.well-known/ready

Performance Tuning and Optimization

Indexing Strategies

Workload	Recommended Index
High recall, <10M vectors	HNSW (all three)
Memory-constrained	IVF_FLAT, PQ/SQ quantization
Billion-scale, disk-based	DiskANN (Milvus)
Hybrid search	Sparse + dense (Qdrant/Weaviate)

Hardware Recommendations

• CPU: 16+ cores for ingestion-heavy workloads.
• RAM: Size at ~1.5x your vector memory footprint for HNSW.
• Storage: NVMe SSD; avoid network-attached storage for hot data.
• GPU: Optional, but a 10x speedup for index builds in Milvus.

Monitoring and Maintenance

Export Prometheus metrics from all three (each exposes a /metrics endpoint). Track query latency p99, recall, memory usage, and segment count. Set up alerts for segment compaction lag — a common source of latency spikes.

Scaling Strategies for Production

Vertical Scaling

Simplest path: add RAM and CPU. Effective up to a point, but diminishing returns and single-point-of-failure risk.

Horizontal Scaling

Add nodes to distribute query and ingestion load. All three support this, but Milvus requires the most planning due to its many components.

Sharding and Replication

flowchart TD
    LB[Load Balancer] --> N1[Node 1 - Shard A]
    LB --> N2[Node 2 - Shard B]
    LB --> N3[Node 3 - Shard C]
    N1 -.replica.-> N1R[Node 1 Replica]
    N2 -.replica.-> N2R[Node 2 Replica]

Use 3x replication for production. Qdrant and Weaviate make this a config flag; Milvus requires specifying replica counts per collection.

Choosing the Right Vector Database for Your Use Case

• Choose Milvus if you’re targeting billions of vectors, need GPU acceleration, or plan a complex multi-tenant platform.
• Choose Qdrant if you want low latency, simple operations, and strong filtering for RAG or recommendations.
• Choose Weaviate if hybrid search and a GraphQL API fit your stack, or you want built-in vectorizer modules.

Conclusion

There’s no universal winner. Milvus wins on scale, Qdrant on operational simplicity and latency, and Weaviate on hybrid search ergonomics. Start with a Docker Compose deployment, benchmark against your data and query patterns, and scale horizontally once you’ve validated the index and hardware profile. The right choice is the one your team can operate confidently at 3 a.m.

FAQ

What is a self-hosted vector database and why would I use one? A self-hosted vector database is a system you deploy and manage on your own infrastructure to store and query high-dimensional vectors for similarity search. You might choose self-hosting for data privacy, cost control at scale, customization, and avoiding vendor lock-in. It’s ideal for RAG applications, recommendation systems, and anomaly detection where you need full control over your data and performance.

Which vector database is easiest to deploy and manage: Milvus, Qdrant, or Weaviate? Qdrant is often considered the easiest to deploy due to its single binary and minimal dependencies. Weaviate offers a straightforward Docker Compose setup with optional modules. Milvus, while more complex due to its distributed architecture, provides Helm charts and operators for Kubernetes that simplify production deployments. For beginners, Qdrant and Weaviate have lower operational overhead.

How do Milvus, Qdrant, and Weaviate perform under high load? Performance depends on index type, dataset size, and hardware. In general, Milvus excels at massive scale (billions of vectors) with its distributed architecture. Qdrant is known for low-latency and high-throughput on moderate datasets. Weaviate offers strong performance with built-in hybrid search. Benchmarks like ANN-Benchmarks show all three can achieve high recall and QPS, but Milvus often leads in scale, while Qdrant and Weaviate are competitive in latency.

Can I scale a self-hosted vector database horizontally? Yes, all three support horizontal scaling. Milvus is designed for distributed deployment with separate components for query, index, and data nodes. Qdrant supports sharding and replication via Raft consensus. Weaviate can be scaled horizontally by adding nodes to a cluster. The approach varies: Milvus uses a microservices architecture, Qdrant uses sharding, and Weaviate uses a shared-nothing cluster.

What hardware should I use for a self-hosted vector database? For optimal performance, use servers with high core-count CPUs (e.g., 16+ cores), ample RAM (64GB+ for millions of vectors), and NVMe SSDs for fast disk I/O. GPU acceleration can significantly speed up index building and querying, especially for Milvus and Weaviate. Network speed is crucial for distributed setups; 10GbE or higher is recommended.

How do I choose between Milvus, Qdrant, and Weaviate for my RAG application? Consider your scale, latency requirements, and feature needs. Choose Milvus for billion-scale datasets and complex distributed setups. Choose Qdrant for low-latency, high-throughput applications with moderate scale and easy deployment. Choose Weaviate if you need built-in hybrid search (vector + keyword) and a GraphQL API. Evaluate based on your team’s expertise and long-term scalability goals.