Self-Hosted LLM Agent Orchestration: LangGraph vs AutoGen vs CrewAI on Kubernetes

Quick Summary (TL;DR) Self-hosting multi-agent systems gives you data sovereignty, cost control, and deep customization—but it demands careful orchestration. LangGraph wins for stateful, cyclic workflows with human-in-the-loop; AutoGen shines for conversational collaboration and code execution; CrewAI excels at role-based team simulation and rapid prototyping. Run them on Kubernetes with vLLM for inference, Redis/PostgreSQL for shared state, and a meta-orchestrator to route tasks between frameworks. This guide walks you through architecture, implementation, and production hardening.

---

Why Self-Host Multi-Agent Orchestration?

Managed platforms are convenient, but they come with trade-offs that matter at scale: per-token pricing that compounds unpredictably, data leaving your perimeter, and limited control over model selection, tool wiring, and execution topology. When your agents touch proprietary data, regulated workloads, or internal APIs, self-hosting stops being a preference and becomes a requirement.

Self-hosting also unlocks capabilities managed platforms rarely expose: custom tool registries, deterministic replay of agent runs, fine-tuned domain models, and cost ceilings enforced at the infrastructure layer rather than the billing layer. You own the observability stack, so you can trace every token, tool call, and state transition.

The cost is operational complexity. You now own inference serving, state persistence, inter-agent networking, and security. This guide gives you a production-ready path through that complexity.

Core Concepts: Agents, Tools, and State

Before comparing frameworks, align on three primitives that every orchestration layer manipulates:

• Agents — An agent is an LLM wrapped with a role, a system prompt, memory, and a decision loop. It reasons about a task, selects tools, and produces output. Agents can be reactive (single turn) or deliberative (multi-step planning).
• Tools — Tools are typed functions the agent can invoke: search, database queries, code execution, HTTP calls. A tool registry centralizes schemas so all frameworks share one source of truth.
• State — State is the accumulated context: conversation history, intermediate results, tool outputs, and control-flow position. State management determines whether your system survives a pod restart, whether you can replay a run, and whether agents can scale horizontally.

The critical insight: state is the hardest part. Stateless agents are easy to scale but lose context. Stateful agents are powerful but require checkpointing, serialization, and a durable store. Every framework below handles this differently.

Framework Comparison: LangGraph vs AutoGen vs CrewAI

Dimension	LangGraph	AutoGen	CrewAI
Core abstraction	Graph of nodes/edges (state machine)	Conversable agents in group chat	Role-based crews with tasks
State model	Built-in checkpointing (thread-level)	Conversation history in memory	Task context passed between agents
Cycles & loops	First-class (explicit edges)	Emergent via conversation	Limited (sequential/hierarchical)
Human-in-the-loop	Native interrupt/resume	Supported via UserProxyAgent	Manual via callbacks
Tool integration	LangChain tool ecosystem	Function calling + code executors	LangChain tools + custom
Best for	Complex stateful workflows	Conversational collaboration, code gen	Role simulation, rapid prototyping
Learning curve	Steep (graph thinking)	Moderate	Gentle
Persistence	Checkpointer (SQLite/Postgres)	Custom (you serialize)	Custom (you serialize)

The pragmatic takeaway: don’t treat these as mutually exclusive. A common production pattern uses LangGraph as the top-level orchestrator, invoking AutoGen group chats for open-ended reasoning and CrewAI crews for well-defined role-based subtasks.

Architecture of a Self-Hosted Orchestration Platform

A layered architecture keeps concerns separated and lets each component scale independently.

flowchart TD
    Client[Client / API Gateway] --> Orch[Meta-Orchestrator<br/>LangGraph]
    Orch --> Router{Task Router}
    Router -->|Stateful workflow| LG[LangGraph Worker]
    Router -->|Conversation| AG[AutoGen Worker]
    Router -->|Role tasks| CW[CrewAI Worker]
    LG --> Registry[Tool Registry]
    AG --> Registry
    CW --> Registry
    LG --> State[(Redis + Postgres<br/>State Store)]
    AG --> State
    CW --> State
    Registry --> Tools[External APIs / DBs / Code Sandbox]
    LG --> LLM[vLLM / TGI Inference Pool]
    AG --> LLM
    CW --> LLM
    LLM --> GPU[GPU Node Pool]
    Orch --> Obs[Observability<br/>OTel + Prometheus + Langfuse]

Each layer has a clear contract: the orchestrator routes, workers execute, the state store persists, the registry resolves tools, and the inference pool serves tokens. This separation is what makes Kubernetes deployment tractable.

Setting Up the Development Environment

Start with a reproducible local stack before touching Kubernetes.

# Create an isolated environment
python -m venv .venv && source .venv/bin/activate

# Install the three frameworks plus supporting libs
pip install langgraph langchain-openai \
            pyautogen \
            crewai \
            redis psycopg2-binary \
            fastapi uvicorn

Point all frameworks at a single OpenAI-compatible endpoint so your code is backend-agnostic. Run vLLM locally:

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-14B-Instruct \
  --port 8000 \
  --tensor-parallel-size 1

Then configure a shared client:

# config.py
LLM_BASE_URL = "http://localhost:8000/v1"
LLM_API_KEY  = "not-needed-locally"
MODEL        = "Qwen/Qwen2.5-14B-Instruct"

Every framework should read from this single config. This is the first step toward avoiding duplication when you scale.

Implementing Agent Communication and State Management

LangGraph: checkpointed state machine

LangGraph’s killer feature is durable state via checkpointers.

from langgraph.graph import StateGraph, END
from langgraph.checkpoint.postgres import PostgresSaver
from typing import TypedDict, Annotated
import operator

class AgentState(TypedDict):
    messages: Annotated[list, operator.add]
    next_step: str

def planner(state: AgentState):
    # LLM decides next action
    return {"next_step": "research"}

def researcher(state: AgentState):
    # Calls tools, appends results
    return {"messages": [{"role": "assistant", "content": "findings..."}]}

builder = StateGraph(AgentState)
builder.add_node("planner", planner)
builder.add_node("researcher", researcher)
builder.add_edge("planner", "researcher")
builder.add_edge("researcher", END)
builder.set_entry_point("planner")

with PostgresSaver.from_conn_string("postgresql://...") as cp:
    cp.setup()
    graph = builder.compile(checkpointer=cp)
    graph.invoke({"messages": []}, config={"configurable": {"thread_id": "run-42"}})

The thread_id gives you replayable, resumable runs—critical for fault tolerance.

AutoGen: conversational handoff

import autogen

config_list = [{"base_url": "http://vllm:8000/v1", "api_key": "x", "model": "Qwen2.5-14B"}]
assistant = autogen.AssistantAgent("assistant", llm_config={"config_list": config_list})
user = autogen.UserProxyAgent("user", code_execution_config={"work_dir": "sandbox"})
user.initiate_chat(assistant, message="Analyze this CSV and plot trends.")

CrewAI: role-based crews

from crewai import Agent, Task, Crew

analyst = Agent(role="Data Analyst", goal="Extract insights",
                backstory="Expert in pandas", llm="openai/Qwen2.5-14B")
task = Task(description="Summarize Q3 revenue", agent=analyst)
Crew(agents=[analyst], tasks=[task]).kickoff()

To share state across all three, serialize each framework’s memory to Redis after every step:

import redis, json
r = redis.Redis(host="redis", port=6379)
def persist(thread_id, payload):
    r.set(f"agent:{thread_id}", json.dumps(payload), ex=86400)

This makes agents stateless pods backed by stateful storage—the key to horizontal scaling.

Integrating Tools and External APIs

Centralize tools in a registry so all frameworks import the same definitions.

# tools/registry.py
from langchain_core.tools import tool

@tool
def query_postgres(sql: str) -> str:
    """Run a read-only SQL query."""
    ...

@tool
def call_internal_api(endpoint: str, payload: dict) -> dict:
    """Call an authenticated internal service."""
    ...

TOOLS = [query_postgres, call_internal_api]

LangGraph and CrewAI consume LangChain tools directly. AutoGen uses function-calling schemas—generate them from the same source:

autogen_tools = [{"type": "function", "function": t.to_openai_function()} for t in TOOLS]

Security rule: never pass raw LLM output to a shell or SQL executor. Validate, sandbox (gVisor/Firecracker), and enforce read-only credentials for database tools.

Deploying to Kubernetes: Scaling and Observability

Inference layer

Deploy vLLM as a Deployment with GPU requests:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args: ["--model", "Qwen/Qwen2.5-14B-Instruct"]
        resources:
          limits: { nvidia.com/gpu: 1 }
        volumeMounts:
        - { name: cache, mountPath: /root/.cache/huggingface }
      volumes:
      - name: cache
        persistentVolumeClaim: { claimName: model-cache }

Use a PersistentVolume for the Hugging Face cache to avoid re-downloading models on every restart.

Agent workers

Agent pods are stateless—scale them with an HPA driven by a queue-length metric rather than CPU:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: agent-workers }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: langgraph-worker }
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric: { name: redis_queue_length }
      target: { type: AverageValue, averageValue: "5" }

Observability

Instrument every agent step with OpenTelemetry and ship traces to Langfuse or Phoenix. Track:

• Tokens per run (cost attribution)
• Tool call latency (bottleneck detection)
• State checkpoint size (storage growth)
• Queue depth (scaling signal)

sequenceDiagram
    participant C as Client
    participant O as Orchestrator
    participant W as Agent Worker
    participant S as Redis/Postgres
    participant L as vLLM
    C->>O: POST /run
    O->>W: dispatch task
    W->>L: inference request
    L-->>W: completion
    W->>S: persist checkpoint
    W-->>O: result
    O-->>C: 200 OK

Security and Cost Considerations

Security checklist:

• Network policies — restrict pod-to-pod traffic; only workers reach the inference pool.
• mTLS via service mesh — encrypt inter-agent communication (Istio/Linkerd).
• Secrets management — use External Secrets Operator or Vault; never bake keys into images.
• RBAC — least-privilege service accounts per workload.
• Input sanitization — validate all tool arguments; sandbox code execution.
• Namespace isolation — separate agents by trust boundary.

Cost control:

• Cap max_tokens and enforce per-run token budgets in the orchestrator.
• Use quantized models (AWQ/GPTQ) to fit more replicas per GPU.
• Scale GPU nodes to zero overnight with Karpenter or cluster-autoscaler.
• Cache embeddings and repeated tool results in Redis.

Conclusion and Next Steps

Self-hosted multi-agent orchestration is a systems problem, not just a framework choice. LangGraph gives you durable, replayable workflows; AutoGen gives you conversational flexibility; CrewAI gives you fast role-based prototyping. Combine them behind a meta-orchestrator, back them with a shared state store and tool registry, and deploy on Kubernetes with GPU-aware autoscaling and end-to-end tracing.

Your next steps: (1) stand up a single-node vLLM + Redis stack locally, (2) port one real workflow to LangGraph with Postgres checkpointing, (3) containerize and deploy to a kind/minikube cluster, then (4) graduate to a managed Kubernetes cluster with GPU nodes and an HPA keyed on queue depth. Start small, instrument everything, and let the state store—not the pods—carry the memory.

---

FAQ

Which framework is best for self-hosted multi-agent orchestration: LangGraph, AutoGen, or CrewAI? It depends on your use case. LangGraph excels at complex, stateful workflows with cycles and human-in-the-loop. AutoGen is ideal for conversational multi-agent collaboration and code execution. CrewAI is best for role-based team simulations and rapid prototyping. Many production systems combine them: use LangGraph for orchestration, AutoGen for agent conversations, and CrewAI for specialized roles.

How do I manage state across multiple agents in a self-hosted setup? Use a centralized state store like Redis or PostgreSQL to persist agent states, conversation history, and tool outputs. LangGraph provides built-in state management with checkpointing. For AutoGen and CrewAI, implement custom state handlers that serialize agent memory to Redis after each step. This enables fault tolerance and horizontal scaling.

Can I run LangGraph, AutoGen, and CrewAI together in one platform? Yes. Build a meta-orchestrator that routes tasks to the appropriate framework. For example, use LangGraph to define the overall workflow, and within a node, invoke an AutoGen group chat or a CrewAI crew. Ensure all frameworks share the same LLM backend and tool registry to avoid duplication.

What are the Kubernetes considerations for scaling LLM agents? Key considerations include using GPU nodes for inference, setting resource requests/limits, implementing horizontal pod autoscaling based on queue length or GPU utilization, using persistent volumes for model caching, and deploying a service mesh for inter-agent communication. Use Helm charts or Operators to manage serving frameworks like vLLM or TGI.

How do I secure a self-hosted multi-agent system? Implement network policies to restrict pod-to-pod communication, use secrets management for API keys, enable RBAC for Kubernetes, and sanitize all tool inputs to prevent injection attacks. Run agents in isolated namespaces and use a service mesh with mTLS for encrypted communication.