1. The Breakdown of Naive RAG in Enterprise Environments
Most developers start building RAG by slicing documents into 500-token chunks, computing OpenAI embeddings, and querying ChromaDB with top-k cosine similarity. In proof-of-concept demos this feels like magic. However, in enterprise environments with nuanced contracts, tables, and evolving policies, naive RAG quickly deteriorates into hallucinations.
The core flaw lies in context fragmentation: when a sentence references 'the aforementioned stipulation in Section 4.2', a 500-token chunk isolated from the header loses its referential anchor, returning irrelevant noise.
2. Hierarchical Chunking & Parent Document Retrieval
To preserve referential integrity, we implement a two-tier indexing strategy. Small semantic chunks (128 tokens) are embedded to maximize search precision against user queries, but each small chunk links back to a broader parent document chunk (1024 tokens).
When the vector index finds the top 3 small chunk hits, the retriever fetches their corresponding parent contexts to supply the model's prompt.
from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain_community.vectorstores import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter
# 1. Define child splitter for precise vector indexing
child_splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=20)
# 2. Define parent splitter for coherent generation context
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=1200, chunk_overlap=100)
retriever = ParentDocumentRetriever(
vectorstore=Chroma(embedding_function=embeddings),
docstore=InMemoryStore(),
child_splitter=child_splitter,
parent_splitter=parent_splitter,
)3. Adding Cross-Encoder Re-Ranking
Bi-encoder vector search is lightning fast, but it computes query and document embeddings independently. A cross-encoder model passes both query and candidate document through transformer attention layers simultaneously, scoring relevance with dramatically higher fidelity.
By retrieving 25 candidates using hybrid search and re-ranking them down to the top 4 using a cross-encoder such as BAAI/bge-reranker-large, hallucination rates drop by over 60%.
4. Conclusion & Production Guardrails
Production RAG is not just a vector database—it is a sophisticated distributed data engineering pipeline. By combining hierarchical parent retrieval, hybrid dense/sparse scoring, and cross-encoder re-ranking, you transform fragile AI prototypes into dependable organizational infrastructure.