How to Build a Self-Hosted AI Assistant with Local RAG and Voice Interface
Quick Summary (TL;DR) This guide walks you through building a private, voice-controlled AI assistant entirely on your own hardware. We’ll combine Ollama (LLM inference), Whisper (speech-to-text), ChromaDB (vector storage), and a Flask backend to create a Retrieval-Augmented Generation (RAG) system that answers questions based on your documents. You’ll end up with a fully functional assistant that keeps your data private, costs nothing in cloud fees, and can be extended to control smart home devices or automate workflows.
Introduction
In an era of data privacy concerns and cloud dependencies, building a self-hosted AI assistant offers full control over your data and customization. While cloud-based assistants like ChatGPT and Alexa are convenient, they come with significant trade-offs: your conversations are processed on third-party servers, and you’re locked into recurring subscription costs.
This guide walks you through creating a voice-controlled assistant that uses local retrieval-augmented generation (RAG) to answer questions based on your documents. We’ll leverage open-source tools: Ollama for LLM inference, Whisper for speech-to-text, ChromaDB as our vector database, and a simple Python frontend. By the end, you’ll have a functional assistant that runs entirely on your hardware—no internet connection required for inference.
Prerequisites and Architecture Overview
Before diving in, ensure you have:
- A machine with at least 16GB RAM (32GB recommended)
- A modern CPU (or GPU for faster inference—NVIDIA with 8GB+ VRAM ideal)
- Ubuntu 22.04 or a similar Linux distribution
- Python 3.10+ installed
- Basic familiarity with the command line
Here’s the architecture we’ll build:
flowchart LR
A[User] -->|Voice Input| B[Whisper STT]
B -->|Text Query| C[Flask Backend]
C -->|Embedding| D[Sentence-Transformers]
D -->|Vector Search| E[ChromaDB]
E -->|Retrieved Chunks| C
C -->|Prompt + Context| F[Ollama LLM]
F -->|Generated Answer| C
C -->|Response| G[Frontend UI]
G -->|Optional TTS| A
The flow is straightforward: user speaks → Whisper transcribes → query is embedded → retrieve relevant chunks from ChromaDB → prompt with context → Ollama generates answer → text-to-speech (optional) via pyttsx3.
Step 1: Setting Up Ollama and Pulling a Model
Ollama makes running LLMs locally incredibly simple. It handles model downloads, quantization, and provides a REST API out of the box.
Install Ollama:
curl -fsSL https://ollama.com/install.sh | sh
Pull a model:
ollama pull llama3.2:3b
This downloads the Llama 3.2 3B parameter model—a good balance of speed and quality for most hardware. For better performance on lower-end machines, consider:
ollama pull mistral:7b-instruct-q4_K_M
Test it:
ollama run llama3.2:3b
You should see a prompt where you can chat with the model. Once verified, exit with /bye.
Verify the API is running:
curl http://localhost:11434/api/generate -d '{
"model": "llama3.2:3b",
"prompt": "Say hello in 5 words"
}'
Step 2: Building the RAG Pipeline with ChromaDB
Now we’ll create a Python script to ingest documents (PDF, text) and store embeddings. We’ll use sentence-transformers/all-MiniLM-L6-v2 for embeddings—a lightweight model that produces 384-dimensional vectors, perfect for local use.
Install dependencies:
pip install chromadb sentence-transformers pypdf
Create ingest.py:
import os
import chromadb
from chromadb.utils import embedding_functions
from sentence_transformers import SentenceTransformer
from pypdf import PdfReader
# Initialize embedding model
embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
# Initialize ChromaDB with persistent storage
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_or_create_collection(
name="documents",
embedding_function=embedding_functions.SentenceTransformerEmbeddingFunction(
model_name="all-MiniLM-L6-v2"
)
)
def extract_text_from_pdf(filepath):
reader = PdfReader(filepath)
text = ""
for page in reader.pages:
text += page.extract_text() or ""
return text
def chunk_text(text, chunk_size=500, overlap=50):
"""Split text into overlapping chunks for better retrieval context."""
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunks.append(text[start:end])
start = end - overlap
return chunks
def ingest_documents(folder_path):
"""Ingest all PDF and TXT files from a folder."""
doc_id = 0
for filename in os.listdir(folder_path):
if filename.endswith(('.pdf', '.txt')):
filepath = os.path.join(folder_path, filename)
if filename.endswith('.pdf'):
text = extract_text_from_pdf(filepath)
else:
with open(filepath, 'r', encoding='utf-8') as f:
text = f.read()
chunks = chunk_text(text)
# Add chunks to ChromaDB
for i, chunk in enumerate(chunks):
collection.add(
documents=[chunk],
ids=[f"doc_{doc_id}_chunk_{i}"],
metadatas=[{"source": filename, "chunk_index": i}]
)
doc_id += 1
print(f"Ingested {filename}: {len(chunks)} chunks")
if __name__ == "__main__":
ingest_documents("./documents") # Create this folder and add your files
Run it:
mkdir -p documents
# Add your PDFs/text files to ./documents
python ingest.py
Step 3: Implementing Voice Recognition with Whisper
OpenAI’s Whisper provides state-of-the-art speech recognition that runs entirely offline. We’ll use the base model for a good speed-to-accuracy ratio.
Install dependencies:
pip install openai-whisper sounddevice numpy scipy
Create voice.py:
import whisper
import sounddevice as sd
import numpy as np
import scipy.io.wavfile as wavfile
import tempfile
import os
# Load Whisper model (base for speed, small for better accuracy)
model = whisper.load_model("base")
def record_audio(duration=5, sample_rate=16000):
"""Record audio from microphone."""
print(f"Recording for {duration} seconds...")
audio = sd.rec(int(duration * sample_rate), samplerate=sample_rate, channels=1, dtype='float32')
sd.wait()
return audio.flatten()
def transcribe_audio(audio, sample_rate=16000):
"""Transcribe audio using Whisper."""
# Save to temp file for Whisper
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
wavfile.write(tmp.name, sample_rate, (audio * 32767).astype(np.int16))
result = model.transcribe(tmp.name)
os.unlink(tmp.name)
return result["text"].strip()
if __name__ == "__main__":
audio = record_audio()
text = transcribe_audio(audio)
print(f"Transcribed: {text}")
Step 4: Creating the Assistant Backend (Flask API)
Now we’ll build a Flask app that ties everything together. The /ask endpoint is where the magic happens: it embeds the query, retrieves relevant chunks, constructs a prompt with context, and calls Ollama.
Create app.py:
from flask import Flask, request, jsonify, render_template
import chromadb
from chromadb.utils import embedding_functions
import requests
import json
import whisper
import sounddevice as sd
import numpy as np
import scipy.io.wavfile as wavfile
import tempfile
import os
app = Flask(__name__)
# Initialize components
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_collection("documents")
embedding_fn = embedding_functions.SentenceTransformerEmbeddingFunction(
model_name="all-MiniLM-L6-v2"
)
whisper_model = whisper.load_model("base")
OLLAMA_URL = "http://localhost:11434/api/generate"
LLM_MODEL = "llama3.2:3b"
def retrieve_context(query, top_k=4):
"""Retrieve relevant document chunks for a query."""
results = collection.query(
query_texts=[query],
n_results=top_k
)
return results['documents'][0]
def generate_answer(query, context):
"""Generate answer using Ollama with RAG context."""
prompt = f"""You are a helpful assistant. Use the following context to answer the question.
If you don't know the answer, say so honestly.
Context:
{context}
Question: {query}
Answer:"""
response = requests.post(OLLAMA_URL, json={
"model": LLM_MODEL,
"prompt": prompt,
"stream": False
})
return response.json()["response"]
@app.route('/')
def index():
return render_template('index.html')
@app.route('/transcribe', methods=['POST'])
def transcribe():
"""Receive audio, transcribe it, and return text."""
try:
# Receive audio file
audio_file = request.files['audio']
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
audio_file.save(tmp.name)
result = whisper_model.transcribe(tmp.name)
os.unlink(tmp.name)
return jsonify({"text": result["text"].strip()})
except Exception as e:
return jsonify({"error": str(e)}), 500
@app.route('/ask', methods=['POST'])
def ask():
"""Receive text query, retrieve context, and generate answer."""
try:
data = request.json
query = data.get('query', '')
# Retrieve relevant context
context_chunks = retrieve_context(query)
context = "\n\n".join(context_chunks)
# Generate answer
answer = generate_answer(query, context)
return jsonify({"answer": answer, "context_sources": context_chunks})
except Exception as e:
return jsonify({"error": str(e)}), 500
if __name__ == "__main__":
app.run(host="0.0.0.0", port=5000, debug=True)
Step 5: Building a Simple Voice-Controlled Frontend
We’ll create a clean, single-page frontend that captures audio, sends it to our backend, and displays the response. We’ll use the MediaRecorder API for audio capture and speechSynthesis for reading answers aloud.
Create templates/index.html:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Local AI Assistant</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
min-height: 100vh;
display: flex;
justify-content: center;
align-items: center;
padding: 20px;
}
.container {
background: white;
border-radius: 20px;
padding: 40px;
box-shadow: 0 20px 60px rgba(0,0,0,0.3);
max-width: 600px;
width: 100%;
}
h1 { text-align: center; color: #333; margin-bottom: 30px; }
.mic-button {
width: 80px;
height: 80px;
border-radius: 50%;
border: none;
background: #667eea;
color: white;
font-size: 30px;
cursor: pointer;
margin: 20px auto;
display: block;
transition: all 0.3s;
}
.mic-button:hover { transform: scale(1.1); }
.mic-button.recording { background: #dc3545; animation: pulse 1s infinite; }
@keyframes pulse {
0% { box-shadow: 0 0 0 0 rgba(220, 53, 69, 0.7); }
70% { box-shadow: 0 0 0 20px rgba(220, 53, 69, 0); }
100% { box-shadow: 0 0 0 0 rgba(220, 53, 69, 0); }
}
.status { text-align: center; color: #666; margin-bottom: 20px; }
.question {
background: #f0f2f5;
padding: 15px;
border-radius: 10px;
margin-bottom: 20px;
}
.answer {
background: #e8f4fd;
padding: 15px;
border-radius: 10px;
margin-bottom: 20px;
white-space: pre-wrap;
}
.hidden { display: none; }
.controls { display: flex; gap: 10px; justify-content: center; }
button.control {
padding: 10px 20px;
border: none;
border-radius: 5px;
background: #667eea;
color: white;
cursor: pointer;
}
button.control:hover { background: #5a67d8; }
</style>
</head>
<body>
<div class="container">
<h1>🎤 Local AI Assistant</h1>
<div class="status" id="status">Click the mic to start speaking</div>
<button class="mic-button" id="micBtn">🎤</button>
<div id="result" class="hidden">
<div class="question"><strong>You:</strong> <span id="questionText"></span></div>
<div class="answer"><strong>Assistant:</strong> <span id="answerText"></span></div>
</div>
<div class="controls">
<button class="control" id="readBtn">🔊 Read Aloud</button>
<button class="control" id="clearBtn">🗑️ Clear</button>
</div>
</div>
<script>
let mediaRecorder;
let audioChunks = [];
let isRecording = false;
let currentAnswer = '';
const micBtn = document.getElementById('micBtn');
const status = document.getElementById('status');
const result = document.getElementById('result');
const questionText = document.getElementById('questionText');
const answerText = document.getElementById('answerText');
micBtn.addEventListener('click', toggleRecording);
async function toggleRecording() {
if (!isRecording) {
// Start recording
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
mediaRecorder = new MediaRecorder(stream);
audioChunks = [];
mediaRecorder.ondataavailable = event => {
audioChunks.push(event.data);
};
mediaRecorder.onstop = processAudio;
mediaRecorder.start();
isRecording = true;
micBtn.classList.add('recording');
micBtn.textContent = '⏹';
status.textContent = 'Recording... click to stop';
} else {
// Stop recording
mediaRecorder.stop();
micBtn.classList.remove('recording');
micBtn.textContent = '🎤';
status.textContent = 'Processing...';
}
}
async function processAudio() {
const audioBlob = new Blob(audioChunks, { type: 'audio/wav' });
const formData = new FormData();
formData.append('audio', audioBlob, 'recording.wav');
try {
// Transcribe
const transcribeRes = await fetch('/transcribe', {
method: 'POST',
body: formData
});
const transcribeData = await transcribeRes.json();
const query = transcribeData.text;
questionText.textContent = query;
result.classList.remove('hidden');
// Get answer
status.textContent = 'Thinking...';
const askRes = await fetch('/ask', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query: query })
});
const askData = await askRes.json();
currentAnswer = askData.answer;
answerText.textContent = currentAnswer;
status.textContent = 'Done! Click mic to ask another question';
} catch (error) {
console.error('Error:', error);
status.textContent = 'Error processing audio. Please try again.';
}
}
document.getElementById('readBtn').addEventListener('click', () => {
if (currentAnswer) {
const utterance = new SpeechSynthesisUtterance(currentAnswer);
speechSynthesis.speak(utterance);
}
});
document.getElementById('clearBtn').addEventListener('click', () => {
result.classList.add('hidden');
currentAnswer = '';
questionText.textContent = '';
answerText.textContent = '';
status.textContent = 'Click the mic to start speaking';
});
</script>
</body>
</html>
Step 6: Testing and Optimization
Test the full system:
python app.py
Open http://localhost:5000 in your browser, click the mic, and ask a question about your documents.
Optimization strategies:
| Area | Adjustment | Impact |
|---|---|---|
| Chunk size | Increase from 500 to 1000 | Better context, slower retrieval |
| Top-K | Increase from 4 to 8 | More context, potential noise |
| Whisper model | Use tiny instead of base | 2x faster, slightly less accurate |
| LLM model | Use llama3.2:1b | 3x faster, less detailed answers |
| GPU acceleration | Enable in Ollama | 10-20x faster inference |
Enable GPU in Ollama:
ollama pull llama3.2:3b
ollama run llama3.2:3b --gpu
Comparison: Self-Hosted vs. Cloud-Based Assistants
| Feature | Self-Hosted | Cloud-Based (ChatGPT, Alexa) |
|---|---|---|
| Privacy | ✅ Complete - data never leaves your machine | ❌ Conversations processed on external servers |
| Cost | 💰 One-time hardware cost, $0/month | 💸 $20+/month subscription |
| Latency | ⚡ 2-5 seconds (local GPU) | 🌐 1-3 seconds (fast servers) |
| Customization | ✅ Full control over models, prompts, integrations | ❌ Limited to provider’s features |
| Maintenance | ⚠️ You handle updates, backups, security | ✅ Provider handles everything |
| Internet dependency | ✅ Works fully offline | ❌ Requires internet connection |
| Hardware requirements | ⚠️ 16GB+ RAM, GPU recommended | ✅ None needed |
Bottom line: Self-hosted wins on privacy, cost, and control. Cloud wins on convenience and zero maintenance.
Conclusion
You now have a fully functional self-hosted AI assistant with a voice interface and local RAG. This setup ensures your data stays private and you have complete control over the technology stack. The system runs entirely on your hardware—no cloud dependencies, no subscription fees, no data leaving your network.
Next steps to expand your assistant:
- Add document types: Support for Word, Markdown, HTML files
- Integrate home automation: Control smart lights, thermostats via MQTT
- Improve the UI: Add chat history, dark mode, mobile responsiveness
- Multi-language support: Use Whisper’s multilingual models
- Custom embeddings: Fine-tune on your domain for better retrieval
The beauty of this setup is its modularity. Swap out Ollama for any OpenAI-compatible API, replace ChromaDB with Pinecone or Weaviate, or change the frontend to a mobile app—the architecture scales with your needs.
Your private AI assistant is ready. What will you teach it first?
FAQ
Q: What hardware do I need to run a self-hosted AI assistant? A: At minimum, 16GB RAM and a modern CPU. For better performance, a GPU with at least 8GB VRAM is recommended. Storage depends on your document corpus and model size—expect 5-10GB for models plus space for your documents.
Q: Can I use any LLM with this setup? A: Yes, Ollama supports many models like Llama, Mistral, and Gemma. You can pull any model from the Ollama library. Just ensure it fits your hardware—smaller quantized models (Q4_K_M) work well on consumer hardware.
Q: How do I add my own documents to the RAG system?
A: Place your documents (PDF, TXT) in the ./documents folder and run python ingest.py. The script will process them and store embeddings in ChromaDB. You can re-run it anytime to add new documents—it will assign new IDs without duplicating existing ones.
Q: Is it possible to use a different speech recognition engine? A: Yes, you can replace Whisper with other engines like Vosk or Coqui STT. However, Whisper is recommended for its accuracy, multilingual support, and fully offline capability. If you need real-time streaming, consider Vosk.
Q: How can I improve the accuracy of the assistant’s answers? A: Several strategies: ensure your documents are well-structured and clean, use a larger chunk size with overlap (e.g., 1000 characters with 100 overlap), increase top-k retrieval (try 6-8), and consider using a larger LLM like Llama 3.1 8B if your hardware supports it. You can also fine-tune the embedding model on your domain for better retrieval.