The Complete Guide to Building a Local LLM Inference Server with vLLM and TensorRT-LLM

Quick Summary: This handbook provides a comprehensive, end-to-end guide to deploying high-performance local LLM inference servers. We cover the architectural foundations, hardware selection, step-by-step setup for both vLLM and TensorRT-LLM, advanced optimization techniques (quantization, batching, memory management), benchmarking methodologies, and production troubleshooting. By the end, you’ll have the knowledge to serve open-source models with near-cloud performance on your own hardware, balancing ease-of-use (vLLM) against raw performance (TensorRT-LLM).

Introduction

The era of cloud-dependent AI is giving way to a new paradigm: local, sovereign, and high-performance inference. As organizations grapple with data privacy regulations, rising API costs, and the need for sub-100ms latency, the appeal of running open-source LLMs on-premises has never been stronger. However, the gap between “running a model” and “serving it efficiently” is vast.

This guide is your technical blueprint for bridging that gap. We will dissect two of the most powerful frameworks in the open-source ecosystem: vLLM, celebrated for its elegant PagedAttention mechanism and developer-friendly Python API, and TensorRT-LLM, NVIDIA’s formidable engine that compiles models into highly optimized CUDA kernels. We’ll move beyond simple “hello world” setups to explore the architectural decisions, memory management strategies, and performance tuning that separate a hobbyist deployment from a production-grade inference server.

Why Local LLM Inference?

Before diving into the pip install commands, it’s crucial to understand the strategic rationale. Local inference isn’t just about avoiding API bills; it’s a fundamental architectural choice.

The Core Advantages:

Data Sovereignty: Sensitive prompts and outputs never leave your infrastructure, ensuring compliance with GDPR, HIPAA, or internal security policies.
Latency Control: Eliminate network round-trips. For interactive applications, this can mean the difference between a 50ms and a 500ms response time.
Offline Capability: Your AI services remain operational during cloud outages or network partitions.
Long-Term Cost Efficiency: While upfront hardware costs are significant, the marginal cost per inference approaches zero, making high-volume workloads more economical.

The Trade-offs:

Hardware Procurement: You are responsible for sourcing, maintaining, and upgrading GPUs.
Operational Overhead: You own the monitoring, scaling, and fault tolerance.
Performance Ceiling: Without careful optimization, your local server may underperform a managed cloud service.

Local inference is not universally better; it’s a strategic choice for workloads where control and privacy outweigh operational convenience.

Understanding the Inference Stack

A production inference server is a complex pipeline, not a single monolithic process. To optimize it, you must understand its components.

flowchart LR
    A[Client Request] --> B[API Gateway / Load Balancer]
    B --> C[Request Queue]
    C --> D[Scheduler]
    D --> E[Batch Manager]
    E --> F[GPU Compute: Model Forward Pass]
    F --> G[KV Cache Memory Manager]
    G --> F
    F --> H[Sampler / Token Decoder]
    H --> I[Response Aggregator]
    I --> J[Client Response]
    
    style F fill:#76b900,stroke:#333,stroke-width:2px
    style G fill:#76b900,stroke:#333,stroke-width:2px

Request Handling & Tokenization: Raw text is converted to token IDs and batched.
Scheduling: The scheduler decides which requests to process when. This is where batching strategies come into play.
The Forward Pass: The core Transformer computation on the GPU. This is where the model weights are applied.
KV Cache: The hidden states (Key and Value tensors) for each token are stored here to avoid recomputation. Managing this cache is the single most critical factor in inference performance.
Sampling & Decoding: The model’s logits are converted into text tokens.

The Core Architectural Difference

vLLM’s PagedAttention: This is a game-changer. It manages the KV cache in fixed-size blocks (pages) rather than a contiguous memory chunk. This eliminates memory fragmentation, allowing the cache to use nearly 100% of available VRAM. It enables continuous batching where requests are dynamically added/removed from the batch at the token level.
TensorRT-LLM’s In-Flight Batching: This achieves similar efficiency through a different mechanism. It uses a static, pre-planned memory pool but optimizes scheduling at the kernel level. Requests are scheduled in a “wave” that can be interrupted and resumed, maximizing GPU occupancy.

Both achieve high throughput, but they approach the problem from opposite ends: vLLM with a dynamic, memory-flexible scheduler; TensorRT-LLM with a static, compute-optimized engine.

Hardware Requirements and Selection

Your GPU is the engine of your inference server. Choosing wrong means either running out of memory or wasting money on idle compute.

The Golden Rule: The model must fit in VRAM, plus room for the KV cache and overhead. As a rule of thumb, you need 1.5x to 2x the model size in FP16 to run comfortably.

Model Size	FP16 VRAM (Model)	Recommended GPU (Consumer)	Recommended GPU (Datacenter)
7B (e.g., Llama 3, Mistral)	~14 GB	RTX 3090 / 4090 (24 GB)	A10G / L4 (24 GB)
13B	~26 GB	2x RTX 3090 (NVLink)	A100 40GB
70B	~140 GB	N/A (Requires multiple GPUs)	4x A100 80GB / 2x H100 80GB

Beyond VRAM:

Memory Bandwidth: This is the true bottleneck for LLM inference. The GPU must stream weights and KV cache from VRAM to compute cores. A GPU with higher bandwidth (e.g., H100’s 3.35 TB/s vs. RTX 4090’s 1.01 TB/s) will generate tokens much faster.
Compute Capability: For FP8 and other modern quantization, you need Ada Lovelace (RTX 40-series) or Hopper (H100) architectures. Older Ampere GPUs are limited to INT8.
CPU & RAM: You need a capable CPU to handle tokenization, scheduling, and the Python overhead. A minimum of 64GB of system RAM is recommended for loading and shuffling models.
Storage: Use NVMe SSDs. Loading a 70B model from a SATA SSD can take minutes, adding to cold-start latency.

Setting Up vLLM

vLLM is the fastest path to a production-grade server. Its Python-first design and OpenAI-compatible API make it a developer favorite.

Installation & Basic Server

# Create and activate a virtual environment (Python 3.9+)
python -m venv vllm-env
source vllm-env/bin/activate

# Install vLLM (CUDA 12.1+)
pip install vllm

Once installed, you can start a server with a single command:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --host 0.0.0.0 \
    --port 8000 \
    --gpu-memory-utilization 0.90

This command does the following:

Loads the model in FP16.
Allocates 90% of VRAM for the model and KV cache.
Starts an OpenAI-compatible HTTP server on port 8000.

The Python API (For Custom Applications)

For more control, you can use the LLM class directly:

from vllm import LLM, SamplingParams

# Initialize the engine
llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.2", tensor_parallel_size=1)

# Define sampling parameters
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=512)

# Generate outputs
outputs = llm.generate(["Explain quantum computing in simple terms."], sampling_params)

for output in outputs:
    print(output.outputs[0].text)

Key Configuration Flags for vLLM:

--max-num-seqs: The maximum number of sequences to process in a single batch (default: 256).
--max-model-len: The maximum sequence length (context + generated tokens). Setting this too high wastes memory.
--quantization: Specify awq, gptq, or fp8 to load quantized models.

Setting Up TensorRT-LLM

TensorRT-LLM is a different beast. It requires a build phase where the model is compiled into a TensorRT engine optimized for your specific GPU. This process is slower to set up but yields the absolute best performance.

The Build Process

# Clone the repository
git clone https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM

# Install dependencies (requires Docker for best results)
make -C docker build

# Enter the container
make -C docker run

# Inside the container, download and build the model
python3 examples/llama/build.py \
    --model_dir meta-llama/Meta-Llama-3-8B-Instruct \
    --dtype float16 \
    --use_gpt_attention_plugin float16 \
    --use_gemm_plugin float16 \
    --output_dir ./engine_output

This generates a .engine file—a highly optimized, GPU-specific binary.

Deployment with Triton

While you can write a Python C++ runtime, the standard production deployment is via Triton Inference Server. This adds a layer of complexity but provides enterprise-grade features like dynamic batching, model ensemble, and metrics.

# Start Triton with the TensorRT-LLM backend
docker run --gpus all --shm-size=1g \
    -v ./engine_output:/models/llama/1 \
    -v ./triton_config:/models/llama \
    nvcr.io/nvidia/tritonserver:24.05-trtllm-python-py3

The Trade-off: You trade the simplicity of pip install vllm for a Docker-based, multi-step build process. In return, you get a server that is often 10-20% faster in raw token generation throughput.

Optimizing Performance: Quantization

Quantization is the art of shrinking model weights to fit more into memory and speed up computation. It’s the single most impactful optimization you can make.

Technique	Precision	VRAM Reduction	Speedup	Accuracy Impact	Hardware Support
FP16	16-bit	Baseline	Baseline	None	All
INT8	8-bit	~50%	~30%	Low	Ampere+
FP8	8-bit float	~50%	~40%	Very Low	Hopper, Ada
INT4 (AWQ/GPTQ)	4-bit	~75%	~60%	Moderate	All (via kernels)

Implementation in vLLM

# For an AWQ-quantized model
python -m vllm.entrypoints.openai.api_server \
    --model TheBloke/Llama-2-7B-Chat-AWQ \
    --quantization awq

Implementation in TensorRT-LLM

# Build with INT4 AWQ
python3 examples/llama/build.py \
    --model_dir meta-llama/Meta-Llama-3-8B-Instruct \
    --quant_ckpt_path ./awq_weights.pt \
    --use_weight_only \
    --weight_only_precision int4_awq

Expert Tip: Always benchmark accuracy on your specific task after quantization. A 1% drop in MMLU might be acceptable, but a 5% drop in a legal document summarization task is not.

Optimizing Performance: Batching and Throughput

The GPU is a throughput machine. To keep it busy, you must batch requests.

Static Batching: Wait for N requests to accumulate, then process them together. Simple but inefficient (GPU idles while waiting).
Dynamic Batching: Add requests to the batch as they arrive, up to a maximum size. Better, but requests that finish early still occupy batch slots.
Continuous Batching (vLLM) / In-Flight Batching (TRT-LLM): This is the gold standard. The scheduler checks after every token generation step if any sequence is complete. If so, it evicts that sequence and adds a new one mid-generation. This keeps the GPU at near-100% utilization.

Configuration Parameters:

Max Batch Size: Higher is better for throughput, but increases latency for individual requests and consumes more KV cache.
Max Wait Time: In dynamic batching, how long to wait for a batch to fill up before processing. A balance between latency and throughput.

Recommendation: Set your max batch size as high as your latency budget and VRAM allows. Monitor GPU utilization; if it’s below 80%, increase the batch size.

GPU Memory Management

Memory is the battlefield of LLM inference. Mismanagement leads to OOM errors or, worse, silent performance degradation.

The KV Cache Problem

For a 7B model, the KV cache for a single 2048-token sequence can consume over 1GB of VRAM. For a 70B model with a 4096-token context, it’s over 10GB. This cache is dynamic and grows with each token generated.

vLLM’s Approach: PagedAttention

flowchart TD
    A[Logical KV Cache for Sequence 1] -->|Maps to| B[Physical Block 1]
    A -->|Maps to| C[Physical Block 3]
    A -->|Maps to| D[Physical Block 7]
    
    E[Logical KV Cache for Sequence 2] -->|Maps to| F[Physical Block 2]
    E -->|Maps to| G[Physical Block 4]
    
    H[Free Memory Pool] --> I[Block 5]
    H --> J[Block 6]
    H --> K[Block 8]

Instead of allocating one contiguous chunk of memory, vLLM allocates fixed-size blocks (e.g., 16 tokens per block). The logical sequence maps to non-contiguous physical blocks. This eliminates fragmentation, allowing you to use up to 90%+ of VRAM without OOM errors.

TensorRT-LLM’s Approach: Static Planning

TensorRT-LLM calculates the maximum possible KV cache size at build time based on your max-batch-size and max-sequence-len. It allocates this entire block upfront. This is less flexible but allows for more aggressive kernel-level optimizations since the memory addresses are known in advance.

Multi-GPU: Tensor Parallelism

For models that don’t fit on one GPU, you split the model across GPUs. Each GPU holds a slice of the weights and computes a portion of the attention heads. The GPUs communicate via NVLink or PCIe.

# vLLM
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-70B-Instruct \
    --tensor-parallel-size 4

Warning: Tensor parallelism adds communication overhead. It’s only worth it if the model cannot fit on a single GPU.

Benchmarking Your Server

You cannot optimize what you cannot measure. Benchmarking is non-negotiable.

Key Metrics

Throughput: Requests per second (RPS) or tokens per second (TPS).
Time to First Token (TTFT): The time from request submission to the first token output. Critical for interactive chat.
Time Per Output Token (TPOT): The inter-token latency. Lower is better for streaming.
Latency Percentiles (p50, p95, p99): Ensure your server isn’t just fast on average, but consistent.

Tools and Scripts

vLLM Built-in Benchmark:

python -m vllm.benchmark.benchmark_throughput \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --backend vllm \
    --input-len 512 \
    --output-len 256 \
    --num-prompts 100

Triton’s perf_analyzer:

perf_analyzer -m llama -u localhost:8000 \
    --concurrency-range 1:16 \
    --input-data ./payload.json \
    --measurement-mode time_windows \
    --measurement-interval 10000

Interpreting Results

If your TTFT is high, your batching is too aggressive, or the model is too large for the GPU (thrashing). If your TPOT is high, your GPU memory bandwidth is the bottleneck. If your p99 latency is much higher than p50, you have scheduling spikes.

Production Edge Cases and Troubleshooting

Production is where theory meets reality. Here are the common pitfalls and how to survive them.

1. Out-of-Memory (OOM) Errors

Symptom: CUDA OOM or torch.cuda.OutOfMemoryError.
Causes: KV cache overflow, batch size too large, memory fragmentation (less likely with vLLM).
Fixes: Reduce --max-num-seqs, reduce --max-model-len, enable --swap-space (vLLM) to offload KV cache to CPU RAM, or use quantization.

2. Request Timeouts

Symptom: Clients receive 504 Gateway Timeout.
Causes: Long prompt prefill (processing the input), or a queue backlog.
Fixes: Implement a separate endpoint for prefill vs. decode, increase timeout thresholds, or scale horizontally behind a load balancer.

3. The “Cold Start” Problem

Symptom: First request after startup is extremely slow.
Cause: The model is being loaded from disk to VRAM.
Fixes: Preload the model (vLLM: --load-format safetensors), or use a keep-alive mechanism to send a “warm-up” request periodically.

4. Monitoring and Observability

You need to see inside the box.

flowchart LR
    A[vLLM Server / Triton] -->|/metrics endpoint| B[Prometheus]
    B --> C[Grafana Dashboard]
    
    D[GPU Metrics: nvidia-smi] -->|Node Exporter| B
    
    E[Application Logs] --> F[ELK Stack / Loki]

Both vLLM and Triton expose Prometheus metrics. Set up alerts for:

GPU Utilization < 30%: Your batching is inefficient.
VRAM Usage > 95%: You’re at risk of OOM.
p95 Latency > 2x Baseline: Something is wrong.

Conclusion

Building a local LLM inference server is a journey from “it works” to “it works fast.” We’ve traversed the landscape from architectural fundamentals to the gritty details of memory paging and quantization. You now have the knowledge to choose between the developer-friendly agility of vLLM and the raw, uncompromising performance of TensorRT-LLM.

The key takeaway is that there is no single “best” setup. The optimal configuration depends on your hardware, your latency requirements, and your operational capacity. Start with vLLM for its simplicity, benchmark your workload, and only switch to TensorRT-LLM if you hit a hard performance ceiling. The power to run world-class AI models is in your data center—now you know how to unleash it.

FAQ

Q: What is the difference between vLLM and TensorRT-LLM? A: vLLM is an open-source Python library known for its ease of use and PagedAttention mechanism, which efficiently manages KV cache memory. TensorRT-LLM is NVIDIA’s proprietary engine that compiles models into highly optimized TensorRT engines, offering lower latency and higher throughput but requiring more setup effort.

Q: How much VRAM do I need to run a 7B model? A: For a 7B model in FP16, you need at least 14GB of VRAM, but 24GB is recommended to accommodate KV cache and overhead. With quantization (INT8 or INT4), you can reduce VRAM usage to around 7-10GB.

Q: Can I use multiple GPUs for inference? A: Yes, both vLLM and TensorRT-LLM support tensor parallelism, which splits the model across multiple GPUs. This is essential for models larger than a single GPU’s VRAM, such as 70B models.

Q: What is continuous batching and why is it important? A: Continuous batching allows new requests to be added to the batch as soon as a previous request finishes, maximizing GPU utilization. It reduces idle time and improves throughput compared to static batching.

Q: What are the common causes of out-of-memory (OOM) errors? A: OOM errors often occur when the KV cache grows too large, the batch size is too big, or the model is too large for the GPU. Solutions include reducing batch size, using quantization, enabling memory swapping, or using multi-GPU tensor parallelism.