Self-hosting#

The entire platform runs on your infrastructure — embeddings, generation, and vector storage included. No data leaves your network.

Minimal stack (single node)#

docker compose up -d
# api        → FastAPI on :8000
# qdrant     → vector store on :6333
# ollama     → local LLM serving on :11434

This footprint handles development and small teams: local sentence-transformers embeddings, Ollama generation, Qdrant storage.

Production topology#

Scale each tier independently:

• API — stateless replicas behind a load balancer; scale on concurrent SSE streams
• Retriever + reranker — the reranker wants a GPU; scale its replicas separately from BM25/fusion
• LLM serving — vLLM with continuous batching on a dedicated GPU pool
• Qdrant — sharded collections with replication; snapshot before bulk re-ingestion
• Workers — queue-driven ingestion; scale on queue depth

See the interactive architecture diagram for how the pieces connect.

Configuration#

Everything is environment-driven — one settings object, no scattered config:

| Variable | Purpose | | --- | --- | | VECTOR_BACKEND | qdrant / chroma / pinecone | | EMBEDDING_PROVIDER | Embedding model selection | | RETRIEVAL_MODE | hybrid / dense / sparse | | GENERATION_PROVIDER | ollama / vllm / hosted APIs | | INGEST_MAX_RETRIES | Retry budget before quarantine |