RAG, fine-tuning, or long context — which should I use for an enterprise knowledge base?

They are not replacements for each other. RAG fits scenarios where answers come from existing documents and need traceable citations (contract review, policy Q&A, product docs): low cost, easy to update, hallucination-controllable. Fine-tuning fits fixed output style and format, but cannot inject new knowledge and requires retraining on every update. Long context fits single long-document Q&A but exceeds the window on multi-document aggregation and cost scales with length. For the vast majority of enterprise knowledge-base scenarios, RAG is the first choice — fine-tuning fixes expression style, not knowledge gaps.

How do I choose between Qdrant, Milvus, pgvector and Elasticsearch?

Decide by team and existing infrastructure. If you already run PostgreSQL and have under ~10M vectors, pgvector starts with zero new components. For large-scale vector search with HNSW and filtered queries, Qdrant is the fastest to adopt and simplest to operate. For billions of vectors and distributed scale, choose Milvus. If you need unified full-text + vector search and already run Elasticsearch, ES is a solid fit. Do not jump straight to a distributed vector DB — most enterprise knowledge bases are a few million chunks, and single-node Qdrant is enough.

What are the three most common causes of poor RAG retrieval quality?

First, naive chunking — splitting by fixed character count cuts semantic units in half and fragments retrieval; the right approach is splitting by heading hierarchy and semantic boundaries. Second, missing metadata — without source, date, and permission filtering when multiple documents coexist in one knowledge base, retrieval results contaminate each other. Third, vector-only search — proper nouns, clause numbers, and abbreviations (contract clause 4.2, model names like Qwen3-30B-A3B) are recalled unreliably by embeddings and require keyword search alongside vectors.

What hardware does an on-prem RAG knowledge base need?

It depends on the model and concurrency, not on knowledge-base size. For 7B-8B models, a single RTX 4090 (24G) with AWQ 4-bit quantization serves dozens of concurrent users. 32B-class MoE models (e.g. Qwen3-30B-A3B) need 48G+ VRAM. 70B dense models require A100/H20 or dual cards. Vector retrieval runs on CPU memory, not GPU — the deciding factor for VRAM is model size and concurrency. Use an LLM sizing calculator to estimate precisely from model, quantization, context length, and concurrency.

How do I evaluate a RAG system after launch?

Split into three layers. Retrieval: build 100+ real high-frequency questions with gold answers, measure recall and MRR; below 80% recall, fix retrieval before touching generation. Generation: sample 50-100 items, score citation accuracy and completeness with human review or LLM-as-judge; answers must carry citations and citations must be real. Business: track follow-up rate (high means first answers miss) and adoption rate after launch; feed frequently-missed questions back into the eval set weekly to close the loop.

Enterprise Private Knowledge Base with RAG: From Architecture Selection to Retrieval Quality (2026 Guide)

RAG is the most common enterprise AI use case. This guide breaks down the full private knowledge-base Q&A pipeline: RAG vs fine-tuning vs long-context decisions, vector DB selection (Qdrant/Milvus/pgvector), chunking and metadata design, hybrid search and reranking, on-prem deployment cost, and post-launch evaluation. Includes a reusable checklist and the most common pitfalls. [See the 6-step RAG checklist →]

Bottom line first

Across the enterprise AI projects we have seen over the past two years, the most frequent and most cost-effective use case is private knowledge-base Q&A: policy Q&A, contract review, product documentation, support knowledge bases, and industry-standards retrieval. The reason is simple — most institutional knowledge lives in documents, and document Q&A is exactly what RAG does best.

But RAG outcomes vary wildly: some teams ship a Qdrant container plus an open-source model in a day and get great results; others tune for months and still get poor retrieval. The difference is not the model — it is knowledge engineering: chunking, metadata, retrieval strategy, and evaluation loops. This guide walks through the full 0-to-1 pipeline: decision boundaries, selection, copy-paste-ready configuration, and an evaluation checklist.

1. Decide first: RAG, fine-tuning, or long context?

Many teams pick wrong at step one. The fit boundaries:

Option	Fits	Does not fit	Cost
RAG	Answers from existing docs, needs citations	Style/format must be internalized	Low, easy to update
Fine-tuning	Fixed output format, domain expression style	Injecting new knowledge	High, retrain on each update
Long context	Single long-document Q&A	Multi-document aggregation	Scales with length

Rule of thumb: knowledge lives in documents and answers must be traceable → RAG; only style and format must be fixed → fine-tuning; a single document of tens of thousands of words → long context. For the vast majority of enterprise knowledge bases, RAG is the first choice. Fine-tuning cannot fix knowledge gaps — you cannot “train the knowledge base into” the model, and even if you could, every update would require retraining.

2. Architecture overview: the full RAG pipeline

A production RAG system has five stages; skipping any of them surfaces in production:

Document intake → parse & clean → chunking → embed → vector store
                                ↓
User question → query rewrite → hybrid search (vector + keyword) → rerank → context assembly → generation → cited output

Most teams spend their effort on “embed → generation” and neglect the ingestion side (parse/clean, chunking, metadata) and the retrieval side (query rewrite, hybrid search, reranking) — yet these two sides set the ceiling on retrieval quality. The model is only the last stage.

3. Vector DB selection: do not start with distributed

Option	Strength	Fits	Watch out
pgvector	Zero new components, reuse existing PG	Already on PostgreSQL, <10M vectors	Filtered queries slower
Qdrant	Fastest to adopt, simplest ops, strong HNSW + filters	First choice for a standalone vector DB	Single node is enough to start
Milvus	Distributed, billion-scale	Huge data, multiple replicas	Many components, heavy ops
Elasticsearch	Full-text + vector in one, native hybrid	Existing ES infrastructure	Vector performance below dedicated DBs

Pragmatic advice: most enterprise knowledge bases stay under a few million chunks — single-node Qdrant is enough. Do not pay for scale you do not have. The selection criterion is not “which is strongest” but “which has the least friction with your existing team and infrastructure.”

4. Chunking and metadata: the first lever on retrieval quality

Chunking

The most common mistake is hard-splitting by fixed character count (e.g. 500 chars per chunk), cutting semantically complete paragraphs in half so retrieval returns fragments. Production-grade practice:

Split by document structure: split into semantic blocks by heading hierarchy (Markdown headings / sections) first, then refine oversized blocks by paragraph — keep the heading text, it is a strong retrieval signal.
Overlap: overlap adjacent chunks by 50-100 chars so key sentences never land exactly on a boundary.
Chunk size bounds: 200-1000 chars per chunk. Too short loses context, too long dilutes retrieval precision.
Parent-child chunking (advanced): retrieve with small chunks, feed the containing parent block as context — balances precision with semantic completeness.

Metadata

The most underrated piece. Every chunk should carry at least:

Source: document ID, title, URL (required for citation traceability)
Hierarchy path: section path for locating answers
Date: policies have versions; retrieval must filter by time
Permission/department: knowledge from different departments must not cross-pollinate

With metadata, retrieval can filter — the 2024 attendance policy must never recall the 2022 edition. Without metadata filtering, multiple documents in one knowledge base contaminate each other’s retrieval results.

5. Retrieval strategy: keyword search is mandatory, not optional

Pure vector retrieval has a fatal weakness in knowledge-base scenarios: proper nouns, clause numbers, and abbreviations are recalled unreliably. “Clause 4.2”, “Qwen3-30B-A3B”, internal codenames — embeddings discriminate poorly on such tokens. Production-grade retrieval is hybrid:

Vector search: semantic recall — handles “asked a different way”
Keyword search (BM25): exact match recall — handles numbers/proper nouns
Reranker: after merging, rerank with a cross-encoder to surface what truly matters

Recommended config: BM25 + vector each recall Top 50, fuse, then rerank to Top 5-8. Rerankers are small in VRAM (a few GB) but deliver outsized retrieval gains — the highest ROI line item in the whole pipeline.

Also add query rewriting: a user asking “is that policy from last time still valid?” will fail direct retrieval. Rewrite the query with a small model (“policy validity 2026”) before searching, and hit rate improves noticeably.

6. On-prem deployment and cost: the model is not as scary as it looks

Private knowledge bases almost always require data to stay inside the network. Deployment splits into two tiers:

Generation tier (LLM)

7B-8B single node: RTX 4090 (24G) with AWQ 4-bit quantization serves dozens of concurrent users; quality is sufficient for knowledge-base Q&A
32B MoE (Qwen3-30B-A3B etc.): 48G+ VRAM recommended; quality approaches 70B dense models
70B dense: A100/H20 or dual cards — only when answer quality is a hard requirement

Retrieval tier (vector DB + reranker)

Good news: vector retrieval does not use GPU — it uses memory. A few million chunks of vectors fit in a few GB of RAM, CPU is fine. What really determines VRAM is model size and concurrency, not knowledge-base scale — many people get this backwards.

For hardware budgets, estimate precisely from four parameters — model, quantization, context length, concurrency — with our on-prem LLM sizing calculator, which outputs VRAM, GPU options, budget, and ready-to-use vLLM launch flags in minutes.

7. Post-launch evaluation: make it a mechanism, not a one-off

The biggest RAG failure is launching without an eval set, relying on “it feels fine.” Production-grade practice:

Layer 1: Retrieval evaluation

Build 100+ real high-frequency questions with gold answers
Measure recall (is the gold answer in the retrieved set?) and MRR (how high is it ranked?)
Below 80% recall, do not touch generation — with broken retrieval, no model can answer correctly

Layer 2: Generation evaluation

Sample 50-100 items, score with humans or LLM-as-judge: citation accuracy, completeness, hallucination rate
Citations are the core value of RAG — answers must carry citations, and the citations must be real

Layer 3: Business evaluation

Track follow-up rate (users asking again after the first answer — high means first answers miss) and adoption rate
Review weekly; feed frequently-missed questions back into the eval set to close the iteration loop

8. Four common pitfalls

1. The model is everything. A bigger model cannot fix broken retrieval. Fix chunking, metadata, and hybrid search first — a 7B model is enough.

2. Fixed-character chunking. Cutting semantic blocks apart fragments retrieval. Splitting by heading hierarchy and semantic boundaries is the highest-ROI step.

3. Vector-only search. Proper nouns and numbers will collapse. Hybrid search + reranking is the baseline, not a bonus.

4. Shipping without evaluation. “It feels fine” gets exposed by users’ follow-ups. A 100-item eval set, three evaluation layers, and weekly feedback are sustainable quality assurance.

Closing: the RAG moat is knowledge engineering, not the model

Look across every stage above: RAG bottlenecks are never “do we have a model” but how finely the knowledge engineering is done — document parsing, chunking, metadata design, retrieval composition, and evaluation loops. This is exactly the work our teams get commissioned for most often in AI engineering. Our AI engineer augmentation service covers RAG and knowledge bases end to end — architecture, selection, build, launch, and iteration — for companies that lack a person who can take RAG to production grade.

Related reading:

AI Agent Workflow Orchestration — the agent layer on top of RAG: tool calling, orchestration architectures, hallucination control

AI for SMEs: Three Real Deployment Stories — RAG in contract review and other real scenarios

Observability for AI Applications — how to observe retrieval and generation quality after launch

Capacity Planning & Performance Testing — concurrency and capacity design for knowledge bases

How to Find a Reliable Technical Partner — a vendor evaluation checklist

On-prem LLM Sizing Calculator — precise hardware budget from model/quantization/context/concurrency

Need architecture, selection, or build capacity for a private RAG knowledge base? Contact us for a free assessment.