Self-Hosted Real-Time Speech-to-Text and Translation Pipeline with Whisper and NLLB on Kubernetes

Quick Summary (TL;DR)

Run Whisper (ASR) and NLLB-200 (translation) as GPU-backed microservices on Kubernetes.

A WebSocket gateway ingests audio, streams chunks via gRPC, and returns partial transcripts + translated text with <500ms target latency.

Use separate GPU node pools, HPA/KEDA autoscaling, and NetworkPolicies for privacy-first, production-grade operation.

Keep audio entirely inside your cluster — no third-party APIs, no data egress.

Introduction: Why Self-Host Real-Time Speech-to-Text and Translation?

Cloud speech APIs are convenient, but they come with three recurring problems: latency, cost at scale, and data governance. When you stream raw audio to a third-party endpoint, you lose control over where that audio lives, how long it’s retained, and who can subpoena it. For healthcare, legal, finance, and government workloads, that’s often a non-starter.

Self-hosting Whisper for automatic speech recognition (ASR) and NLLB-200 (No Language Left Behind) for translation gives you a pipeline that runs entirely inside your own infrastructure. You get predictable latency, no per-minute billing surprises, and full custody of the data. The trade-off is operational complexity — which is exactly what Kubernetes is good at absorbing.

This guide walks you through a production-ready deployment: model serving, a streaming gateway, GPU autoscaling, and the security controls that make the whole thing defensible.

Architecture Overview: Components and Data Flow

The pipeline has four logical layers: ingestion, ASR, translation, and delivery. Decoupling them lets you scale each independently and swap models without touching the others.

flowchart LR
    A[Client / Browser] -->|WebSocket audio chunks| B[Streaming Gateway]
    B -->|gRPC stream| C[Whisper ASR Service]
    C -->|partial + final transcript| B
    B -->|gRPC unary/batch| D[NLLB Translation Service]
    D -->|translated text| B
    B -->|WebSocket JSON| A
    C -.-> E[(Model Store / MinIO)]
    D -.-> E
    B --> F[Prometheus / OTel]
    C --> F
    D --> F

Key design decisions:

WebSocket for ingestion — browsers and mobile clients speak it natively, and it supports bidirectional streaming.
gRPC between services — lower overhead than REST, native streaming, and strongly typed protobuf contracts.
Separate GPU pools — Whisper and NLLB compete for VRAM; isolating them prevents OOM cascades.

Prerequisites and Cluster Setup

Before deploying, confirm you have:

A Kubernetes cluster (v1.27+) with the NVIDIA device plugin installed.
A GPU node pool with at least 16GB VRAM per model (T4, A10G, L4, or better).
Helm 3, kubectl, and Istio or NGINX Ingress for TLS termination.
Object storage (MinIO or S3-compatible) for model weights.

Label your GPU nodes so workloads land predictably:

kubectl label nodes gpu-node-1 gpu-node-2 workload=asr
kubectl label nodes gpu-node-3 gpu-node-4 workload=nmt

Install the device plugin via Helm:

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm install nvdp nvdp/nvidia-device-plugin \
  --namespace kube-system \
  --set tolerations[0].key=nvidia.com/gpu

Verify schedulable GPUs:

kubectl get nodes -o json | jq '.items[].status.allocatable["nvidia.com/gpu"]'

Deploying Whisper for Streaming ASR

We’ll serve Whisper with Faster-Whisper (CTranslate2 backend) — it’s 4x faster than the reference implementation and supports batched inference. Wrap it in a small FastAPI service exposing gRPC.

Deployment manifest (excerpt):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: whisper-asr
spec:
  replicas: 2
  selector:
    matchLabels: { app: whisper-asr }
  template:
    metadata:
      labels: { app: whisper-asr }
    spec:
      nodeSelector: { workload: asr }
      containers:
        - name: whisper
          image: ghcr.io/your-org/whisper-server:1.0
          args: ["--model", "large-v3", "--compute-type", "float16"]
          resources:
            limits: { nvidia.com/gpu: 1, memory: 20Gi }
            requests: { nvidia.com/gpu: 1, memory: 12Gi }
          readinessProbe:
            httpGet: { path: /healthz, port: 8000 }
            initialDelaySeconds: 30
          volumeMounts:
            - { name: models, mountPath: /models }
      volumes:
        - name: models
          persistentVolumeClaim: { claimName: whisper-models }

Streaming strategy: Whisper is not natively streaming, so we use a sliding-window approach. The gateway maintains a 5-second rolling buffer and re-runs inference every 500ms, emitting partial hypotheses. When a silence threshold is detected (VAD), the buffer is finalized and a final transcript is emitted. This is how you get sub-second perceived latency without a true streaming model.

Deploying NLLB for Real-Time Translation

NLLB-200 distilled 1.3B is the sweet spot for real-time: ~6GB VRAM and roughly 15–30ms per sentence on an A10G. Serve it with CTranslate2 or vLLM for batched throughput.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nllb-nmt
spec:
  replicas: 2
  template:
    spec:
      nodeSelector: { workload: nmt }
      containers:
        - name: nllb
          image: ghcr.io/your-org/nllb-server:1.0
          env:
            - { name: MODEL_ID, value: "facebook/nllb-200-distilled-1.3B" }
            - { name: MAX_BATCH_SIZE, value: "16" }
            - { name: MAX_BATCH_WAIT_MS, value: "20" }
          resources:
            limits: { nvidia.com/gpu: 1, memory: 16Gi }

Batching matters. NLLB’s throughput scales nearly linearly with batch size up to ~32. Set MAX_BATCH_WAIT_MS to 20ms — this trades a tiny latency hit for a large throughput gain when many streams are active.

Language routing: NLLB uses BCP-47-ish codes (eng_Latn, spa_Latn, deu_Latn). Map Whisper’s detected language to the NLLB source code in the gateway, not in the model server, so you can hot-swap language pairs without redeploying.

Building the WebSocket Streaming Gateway

The gateway is the orchestration layer. It handles authentication, per-stream buffering, VAD, and fan-out to ASR and NMT.

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant W as Whisper
    participant N as NLLB
    C->>G: WS connect (JWT)
    G-->>C: 101 Switching Protocols
    loop Every 20ms
        C->>G: binary audio frame (PCM/Opus)
    end
    G->>W: gRPC StreamChunk(buffer)
    W-->>G: partial transcript
    G->>N: Translate(partial, src, tgt)
    N-->>G: translated text
    G-->>C: {partial, translated}
    Note over G,W: On VAD silence → finalize
    G->>W: Finalize()
    W-->>G: final transcript
    G->>N: Translate(final)
    N-->>G: final translated
    G-->>C: {final, translated}

Implementation notes:

Use asyncio (Python) or Tokio (Rust) for the gateway — you’ll be juggling thousands of concurrent sockets.
Decode Opus to 16kHz mono PCM before sending to Whisper; Whisper expects 16kHz.
Attach a stream ID to every gRPC call so responses can be correlated.
Apply backpressure: if the ASR service is saturated, drop partial frames rather than queueing unboundedly.

A minimal gateway skeleton:

async def handle_stream(ws, whisper, nllb, src, tgt):
    buffer = AudioBuffer(seconds=5)
    async for frame in ws:
        buffer.append(decode_opus(frame))
        if buffer.ready_for_partial():
            partial = await whisper.transcribe(buffer.window())
            translated = await nllb.translate(partial.text, src, tgt)
            await ws.send_json({"partial": partial.text, "translated": translated})
        if buffer.silence_detected():
            final = await whisper.finalize(buffer)
            await ws.send_json({"final": final.text, "translated": await nllb.translate(final.text, src, tgt)})
            buffer.reset()

GPU Autoscaling and Resource Management

GPUs are expensive and slow to provision. You want fast scale-up and conservative scale-down.

Use KEDA to scale on a custom metric — active WebSocket sessions per replica:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: whisper-scaler }
spec:
  scaleTargetRef: { name: whisper-asr }
  minReplicaCount: 2
  maxReplicaCount: 10
  cooldownPeriod: 300
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        query: sum(whisper_active_streams) / count(kube_pod_info{app="whisper-asr"})
        threshold: "4"

Cluster Autoscaler should be configured with --scale-down-delay-after-add=10m on GPU node pools to avoid thrashing. Consider spot instances for ASR replicas — Whisper is restart-tolerant if you buffer client-side.

Ensuring Privacy and Security

Privacy is the whole point, so enforce it at every layer:

Layer	Control
Network	`NetworkPolicy` allowing only gateway → ASR → NMT traffic
Transport	mTLS between services (Istio) + TLS for client WebSockets
Auth	JWT validation at the gateway; short-lived tokens
Storage	Encrypted PVCs (`encrypted: true` in StorageClass)
Audit	Structured logs shipped to Loki, no raw audio persisted
Egress	`NetworkPolicy` denying all egress by default

A default-deny egress policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: deny-egress }
spec:
  podSelector: {}
  policyTypes: [Egress]

Never write raw audio to disk. Keep buffers in memory only; if you need debugging, record hashed metadata, not PCM.

Performance Tuning and Monitoring

Instrument everything with OpenTelemetry, export to Prometheus, and dashboard in Grafana. The metrics that matter:

asr_partial_latency_ms — p50/p95/p99 from frame arrival to partial emission.
nmt_batch_size — if it’s consistently 1, you’re wasting GPU cycles.
gpu_memory_used_bytes — alert at 90% to catch leaks before OOMKill.
ws_streams_active — drives autoscaling.
end_to_end_latency_ms — the number your users actually feel.

Tuning checklist:

Set torch.backends.cuda.matmul.allow_tf32 = True for a free ~15% speedup on Ampere+.
Use float16 or bfloat16 — never float32 in production.
Pin models to a local NVMe PVC; loading large-v3 from network storage adds 30–60s cold start.
Batch NLLB aggressively; batch Whisper only across different streams.

Comparison: Whisper vs. NLLB vs. Cloud Services

Criterion	Whisper (self-hosted)	NLLB-200 (self-hosted)	Cloud APIs (AWS/GCP/Azure)
Latency	200–400ms (streaming window)	15–40ms/sentence	300ms–1s
Cost at scale	Fixed GPU cost	Fixed GPU cost	Per-minute, grows linearly
Data residency	Full control	Full control	Vendor-dependent
Model quality	Excellent (large-v3)	Very good (200 langs)	Excellent, opaque
Ops burden	High	High	Low
Customization	Full (fine-tune)	Full (fine-tune)	Limited
Compliance	You own it	You own it	Shared responsibility

Rule of thumb: if you’re processing more than ~50,000 minutes/month, self-hosting usually wins on cost. If you need <100ms latency at global scale with zero ops, cloud APIs still win.

Conclusion and Next Steps

You now have a blueprint for a fully self-hosted, real-time speech-to-text and translation pipeline: Faster-Whisper for ASR, NLLB-200 for translation, a WebSocket gateway for streaming, and Kubernetes primitives for scaling and security. The architecture is model-agnostic — swap Whisper for a streaming model like distil-whisper or a CTC-based system when latency demands it, and upgrade NLLB to a larger variant when quality matters more than speed.

Next steps to harden this further:

Add fine-tuning on domain vocabulary (medical, legal) using LoRA.
Implement speaker diarization upstream of ASR for multi-party calls.
Introduce canary deployments with Istio weighted routing for model updates.
Add end-to-end tracing with OpenTelemetry to catch latency regressions before users do.

FAQ

What hardware is required for real-time Whisper and NLLB inference? For real-time performance, use at least one NVIDIA GPU with 16GB VRAM (T4, V100, or A10G) per model. Whisper large-v2 needs ~10GB; NLLB-200 distilled 1.3B needs ~6GB. In production, run ASR and translation on separate GPU nodes to avoid resource contention and OOM cascades.

How do I handle multiple concurrent audio streams? Use a WebSocket gateway that spawns a dedicated async worker per stream. Each worker buffers audio chunks and sends them to Whisper via gRPC. Scale by deploying multiple gateway replicas behind a Service with session affinity, or decouple ingestion with Redis Streams.

Can I use CPU-only nodes for this pipeline? Yes, but with significant latency trade-offs. Whisper tiny or base models run on CPU at ~2–5x real-time factor — unsuitable for live captions. For real-time, GPU acceleration is strongly recommended. If cost is a concern, use spot GPU instances with graceful shutdown handling.

How do I update models without downtime? Use Kubernetes rolling updates with readiness probes. Deploy new model versions as separate Deployments and switch traffic via a Service selector or Istio VirtualService. For A/B testing, use canary deployments with weighted routing. Store model files on a PersistentVolume or MinIO for fast loading.

What are the latency considerations for real-time translation? End-to-end latency spans audio capture, network transfer, ASR inference, translation inference, and rendering. Aim for <500ms. Optimize with smaller models (Whisper base, NLLB distilled), request batching, gRPC streaming, and emitting partial hypotheses so translation can begin before the final transcript.

How do I secure the pipeline for privacy-sensitive applications? Deploy entirely within your private network, use TLS/mTLS for all communication, and enforce Kubernetes NetworkPolicies to restrict traffic. Avoid external APIs. Encrypt PersistentVolumes at rest, authenticate the WebSocket gateway with JWT, and enable audit logging for all access.

What monitoring tools should I use? Prometheus for metrics (GPU utilization, request latency, error rates), Grafana for dashboards, and OpenTelemetry for distributed tracing. Alert on high latency, GPU memory exhaustion, and pod restarts. Aggregate logs with Loki or Elasticsearch for debugging.

Can I run this on a single-node Kubernetes cluster? Yes, for development or small-scale use — k3s or MicroK8s with a GPU works fine. For production, use a multi-node cluster with separate GPU node pools for high availability and independent scaling of ASR and NMT.