The story
The problem
A legal tech startup needed to answer complex legal questions from a large corpus of case law, contracts, and regulations. Generic LLMs produced confident but incorrect answers, often citing non-existent cases or misinterpreting precedents. The firm could not adopt the tool until the answer quality was reliable and every response was traceable to source documents.
The approach
I built a Retrieval-Augmented Generation system that grounds every answer in retrieved source material. The system uses a hybrid search approach and a validation layer to prevent hallucinations and surface confidence levels to the user.
Architecture
Documents are ingested through a semantic chunking pipeline that respects section boundaries, headings, and cross-references. Chunks are stored as dense vectors in Pinecone and also indexed for keyword search. At query time, the system runs both vector and BM25 retrieval, then reranks the combined results with a cross-encoder model. The top-k chunks are passed to a generation model with a strict prompt that requires citations and a confidence statement. A separate validation agent cross-references the generated answer against the retrieved context to flag unsupported claims.
Frontend and user experience
The React interface shows the answer, a confidence score, and clickable citations that open the referenced document at the relevant passage. Users can inspect the source, which builds trust and makes the tool useful for research rather than just quick answers. Search history and feedback are stored to improve the retrieval model over time.
Results
The system achieved a measured accuracy of 94% on a held-out set of legal questions, up from 67% with a naive vector-only approach. Response times stayed under three seconds for typical queries. The firm deployed the tool to over fifty legal professionals, and source citations became the most-used feature.
What made it hard
Accuracy Requirements
Legal domain requires extremely high accuracy, hallucinations are unacceptable
Document Complexity
Processing complex legal documents with nested structures and cross-references
Latency Optimization
Keeping response times under 3 seconds despite complex retrieval pipeline