Open WebUI, AnythingLLM, or LibreChat: the UI question is a retrieval question

The stack decision people ask about last is the one their users experience first. You’ve got the GPU, you’ve picked the inference server, retrieval works in a notebook — and now somebody actually has to use the thing. So: Open WebUI, AnythingLLM, or LibreChat?

Almost every answer to that question is a feature list, which is exactly how people choose wrong. All three will show you a clean chat window over your own model in an afternoon. What will matter in a year isn’t in the feature list at all.

They agree about more than they disagree

All three are self-hosted, Docker-first, multi-user chat over an OpenAI-compatible endpoint. All three do conversation history, model switching, per-user accounts, streaming, markdown rendering, file upload. Point any of them at the inference server you picked and you’ll have something demo-able before lunch.

That’s why the comparison posts all disagree with each other. Whoever wrote them evaluated whichever one they installed first, found it good — because it is — and stopped.

“Who owns your data” is the wrong question

The framing you’ll usually see is that Open WebUI and AnythingLLM keep your documents while LibreChat doesn’t. It sounds right. It isn’t.

All three take an external vector store. Open WebUI supports pgvector, Qdrant and Milvus — and requires an external one the moment you run more than a single worker. AnythingLLM ships with LanceDB embedded but also speaks pgvector, Chroma, Milvus, Qdrant, Weaviate and Pinecone. LibreChat’s RAG API is a separate LangChain/FastAPI service backed by pgvector. Where the vectors physically live is a config line in every one of them.

The real question is who owns the pipeline: chunking, which embedding model runs, at what dimensionality, whether queries get an instruction prefix, how dense and keyword results get fused, what reranks them — and, the part that bites, whether you can change any of it in eighteen months.

Here’s the test I’d use. Suppose you decide your hybrid search should fuse dense and keyword results with reciprocal rank fusion instead of a weighted blend, because RRF compares ranks rather than scores and so doesn’t need a weight tuned per corpus. Is that a config knob, a plugin, or your own code?

That one question sorts these three cleanly.

Where each one sits

AnythingLLM owns the pipeline by design. You create a workspace, drop documents in, and it chunks, embeds and retrieves. It is the shortest path in this space from zero to private document chat, and for a lot of firms that is the correct trade. The cost is that its retrieval is its retrieval — you’re choosing an embedder and a vector store, not a retrieval strategy.

Fair warning on that paragraph: AnythingLLM is the one of the three whose architecture I’ve read rather than run in production. Weight it accordingly. I’m describing documented design, not scars.

Open WebUI owns the pipeline by default and rents it out. Its native retrieval is more serious than people expect: BM25 alongside dense search, a reranking stage, and real knobs — RAG_HYBRID_BM25_WEIGHT, RAG_TOP_K, RAG_TOP_K_RERANKER, RAG_RELEVANCE_THRESHOLD, RAG_RERANKING_MODEL. You can point its embedding engine at your own vLLM server. And when the knobs run out, Pipelines, Functions and OpenAPI tool servers let you put your own retrieval behind the UI and keep everything else it gives you.

One subtlety before you lean on the external-vector-store option: Open WebUI still embeds the query with whatever embedding model it is configured with. Point it at a collection something else wrote and the query side has to match the document side exactly. Get that wrong and recall degrades without erroring — you’ll get plausible answers to the easy questions and nothing useful on the hard ones, which is the worst failure mode retrieval has.

LibreChat is closest to a pure front end. RAG exists — a separate service you run alongside it — but the centre of gravity is elsewhere: many providers, agents, MCP, a genuinely polished chat surface. If retrieval already lives in a service you own, LibreChat is the one that fights you least, because it isn’t trying to be your retrieval layer in the first place.

I built the same system both ways

Not as an experiment. The second one was supposed to save me work.

I had a document-search stack I liked: hybrid dense + BM25 retrieval, rank-based fusion, a cross-encoder reranker, a multi-step agent that could retrieve again when the first pass came back thin, and a groundedness check before anything reached the user. Then I rebuilt the same concept natively inside a chat UI, to find out how much of it I could get for free.

The answer was about 90%, with almost no custom code. That’s the honest headline and it’s an argument for the native path. Same embedding model. Same reranker model. Hybrid search and reranking already there, with knobs.

The missing 10% wasn’t cosmetic. The agent loop had no equivalent — native retrieval is single-shot, so retrieve, grade, retrieve-again was simply gone. Adjacent-chunk expansion, gone. The groundedness pass, gone. Those were the parts that made the system trustworthy on hard questions, which is to say the parts that mattered.

Other things diverged in ways I had no say in. My rank-based fusion became a weighted ensemble — a legitimate strategy, just not mine, and now with a weight to tune per corpus. I had been truncating embeddings to 1536 dimensions; that turned out to be impossible, because the inference server ignores the dimensions parameter and the host doesn’t truncate client-side, so it stores the full 4096. Not worse quality. Just more storage and more latency, decided for me.

The detail that taught me the most was smaller than any of that. My model emits its reasoning inside <think> tags, but the chat template injects the opening tag into the prompt, so the model only ever emits the closing one — leaving the UI with an unbalanced block that it rendered inline, in the middle of the answer. The fix was a global filter that prepends the missing opening tag to the first streamed token, after which the host’s own collapsible reasoning box works perfectly.

That filter is about fifteen lines and I’m fond of it. But look at what it is: to change how my model’s output was displayed, I had to express the fix in someone else’s plugin idiom, against their renderer’s assumptions. Multiply that by every improvement you’ll want over two years. Then add the version dependency — whether I could set an embedding query prefix at all depended on whether the release I happened to be running exposed the setting.

The other direction sends a bill too, and I’ll be just as plain about it. Running my own backend with a chat client in front of it meant I owned authentication, conversation history, sharing, permissions, file handling, and every piece of chat polish users assume is free because every commercial product has it. That is a great deal of undifferentiated work and it does not end.

So here is the trade with nothing hidden: you either inherit someone else’s retrieval and their release cadence, or you inherit the cost of building chat. There is no third option where you get both. Pick the one you’d rather own for two years.

What I’d actually tell you to run

And the inversion, because a comparison without one is an advertorial: if you’re five people with 200 PDFs, the built-in retrieval in any of these is genuinely fine, and building your own pipeline so you can control the fusion strategy would be malpractice. The pipeline question only starts paying rent when retrieval quality is the thing standing between you and people trusting the system.

The one decision that’s expensive to undo

Not which UI. Not where the vectors live. Whose pipeline computed them.

Embeddings are only reusable if you can reproduce the query side exactly — same model, same dimensionality, same prefix convention, same normalisation. Documents ingested through a UI’s own pipeline are cheap to search inside that UI and surprisingly expensive to take anywhere else, because moving them means either re-embedding the entire corpus or reverse-engineering what was done to it.

So if you’re genuinely unsure, and most people at this stage are, keep the pipeline yours even where the UI could run it. Ingest with your own script into your own store, and let the UI query it. It costs a day now and it keeps the expensive decision open.

The short version

The engine question was how many people at once. The UI question is who owns retrieval. Neither is settled by a feature list, which is why feature lists keep producing different winners.

Answer the fusion test honestly for your own team — knob, plugin, or your own code — and the choice mostly makes itself.

The demo on this site is the second pattern: a thin chat surface over a retrieval service I own, which is why it can show you a citation on every claim rather than a confident paragraph. If you’re deciding which of these to put in front of your own corpus, that’s worth a conversation before you ingest 10,000 documents into something, not after.