Set Up RAG in Open WebUI: Query Your Documents

Open WebUI supports RAG (Retrieval-Augmented Generation): upload docs, ask questions, get answers with citations.

Prerequisites

• Open WebUI running (Docker or pip)
• Ollama or another model connected
• Documents (PDF, TXT, MD, DOCX)

Step 1: Enable RAG

In Open WebUI settings, ensure RAG/document upload is enabled. Some deployments need a vector DB (Chroma, etc.)—check the docs for your setup.

Step 2: Create an Assistant

Create a new Assistant. In the instructions, add: "Answer only from the uploaded documents. Cite the source. If the answer isn't in the documents, say so."

Step 3: Upload documents

In the Assistant or chat, upload files. Open WebUI chunks and indexes them. Wait for indexing to complete.

Step 4: Query

Ask questions in natural language. The model retrieves relevant chunks and generates an answer. Check the citations—click to see the source passage.

Step 5: Refine

If answers are off, try: more specific instructions, different chunk size, or adding metadata to chunks. For large doc sets, consider filtering by date or category.

Use cases

• Internal wiki Q&A
• Contract or policy lookup
• Knowledge base for support
• Research over a document collection

All data stays local if you use Ollama. No cloud upload.