Self-Host OpenRAG: A Complete RAG Stack in One Deploy

Your RAG pipeline returns garbage answers, so you go looking for the culprit. Maybe the embedding model is too small. Maybe the vector database needs different index parameters. Maybe the chunk size is wrong.

Meanwhile the actual problem happened twenty minutes earlier: your PDF parser flattened a financial table into a single line of run-together numbers, and no amount of retrieval tuning recovers information that was destroyed at ingestion.

That's the part of RAG nobody demos, and it's the reason OpenRAG is worth a look.

What's Bundled

OpenRAG is a single-package RAG platform from the Langflow team. Rather than a library you assemble a stack around, it ships the stack:

If you've read our vector database comparisons, you know that choice matters at scale. OpenRAG takes it off the table by picking OpenSearch, which gets you hybrid search (semantic plus BM25 keyword) without wiring two systems together.

Docling Is the Interesting Part

Most RAG tutorials use a naive text extractor. It walks the PDF, pulls out characters in roughly reading order, and hands you a string. For a plain prose document that's fine. For anything real, it's lossy in ways that are invisible until retrieval fails.

Tables lose their row and column relationships. Multi-column layouts interleave. Headers and footers get inlined into body text. Scanned pages produce nothing at all.

Docling does layout analysis instead: it identifies structure, preserves tables as tables, and handles OCR. The chunks that reach your embedding model actually correspond to semantic units.

This is why "we bundled a document parser" is a more meaningful feature than it sounds. The rest of the pipeline can only work with what ingestion hands it.

The Deployment Gotcha

Here's the one that will cost you time. Docling does not run as one of the containers. You start it on the host first:

git clone https://github.com/langflow-ai/openrag.git
cd openrag
uv sync
cp .env.example .env
uv run python scripts/docling_ctl.py start --port 5001

Then bring up the rest:

docker compose up -d

If you skip the Docling step and go straight to docker compose up, the stack starts, the UI loads, and document ingestion fails in a way that looks like an application bug. There's a GPU variant too:

docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d

You'll also need uv and Python 3.13 on the host, which is a heavier prerequisite than a pure-Docker deployment.

Six Services, Six Ports

Only the frontend needs to face users. OpenSearch on 9200 and Langflow on 7860 are administrative surfaces, and Langflow in particular has had a CVSS 9.3 in its history. Keep them internal.

Be Honest About Resources

Upstream calls for 8 GB RAM minimum and 50 GB of free disk. OpenSearch alone is a JVM service with real appetite, and you're running six things.

Elestio's template provisions more headroom than that floor. Either way, this is not a story you fit on a small VM, and that's the trade: you get a complete stack instead of assembling one, and you pay for it in memory.

The Agent Angle

Worth flagging: OpenRAG exposes an MCP server at /mcp over streamable HTTP with API key auth, compatible with Claude Desktop and Cursor. So the same corpus you query through the chat UI is reachable as a tool by your agents. There are also Python and TypeScript SDKs (pip install openrag-sdk, npm install openrag-sdk).

Troubleshooting

Ingestion fails but the UI works. Docling isn't running on the host. Check port 5001.

OpenSearch won't start. OPENSEARCH_PASSWORD has to meet OpenSearch's complexity rules, and the container exits if it doesn't.

Retrieval quality is poor on scanned documents. Confirm OCR actually ran. This is exactly the failure mode Docling exists to prevent, but it needs the right document type detected.

Everything is slow. Check memory before you tune the index. Under the 8 GB floor, OpenSearch and the JVM will thrash.

Langflow flows don't persist. Set LANGFLOW_SECRET_KEY. Without it, encrypted fields don't survive a restart.

Running It

Self-hosting this by hand means six services, a host-level Python dependency, and an OpenSearch instance to keep healthy. Doable, but it's an afternoon.

OpenRAG on Elestio starts at $55/month fully managed, with SSL, automated backups, monitoring and updates handled, across Hetzner, DigitalOcean, Vultr, Linode, Scaleway, Netcup, AWS or your own VM. The price reflects the resource floor above, not a premium on the software, which is open source either way.

If your RAG results have been disappointing and you've been tuning retrieval, try fixing ingestion first. That's the lesson worth taking even if you never deploy this.

Thanks for reading ❤️ See you in the next one 👋