# From Noisy Retrieval to Precision: Building a Production RAG Pipeline for Indian Tax Law

## Background

Indian tax law — GST, Income Tax, circulars, notifications — is dense, cross-referential, and unforgiving of ambiguity. A chatbot that retrieves the wrong section confidently is worse than one that admits it doesn't know. Getting retrieval right is not optional; it's the whole game.

This post documents the full journey of building a production RAG pipeline for a tax Q&A system: data collection, chunking, vector ingestion, hybrid search, reranking, and finally the technique that made the biggest difference — **contextual chunking**.

* * *

## Stage 1: Data Collection and Corpus Preparation

The first step was sourcing and cleaning the raw corpus: GST acts, Income Tax Act sections, CBDT/CBIC circulars, notifications, and FAQs. Each document was parsed and enriched with structured metadata before storage:

*   `source` — act name, circular number, or notification reference
    
*   `section` / `chapter` — structural position within the document
    
*   `effective_date` — relevant for time-bound provisions
    
*   `document_type` — act | circular | notification | FAQ  
    Metadata isn't decoration — it becomes a filter layer during retrieval, letting you scope searches to a specific document type or date range.
    

* * *

## Stage 2: Chunking Strategy (v1 — Naive Overlap)

Standard recursive character splitting with overlap. Chunks of ~400–500 tokens with a 50-token overlap, metadata injected into each chunk's payload.

The intuition behind overlap is sound: if a concept spans a chunk boundary, the overlap ensures neither chunk loses it entirely. In practice, for dense legal text, this assumption breaks. A 50-token overlap cannot carry the subject of a section header that appeared 300 tokens ago.

Ingested into a Qdrant vector store using `BAAI/bge-m3` embeddings (dense-only, 1024 dimensions).

* * *

## Stage 3: Baseline Evaluation

A hand-labelled evaluation set of **100 question–answer pairs** was prepared, covering a representative spread of GST and Income Tax queries — section lookups, rate queries, exemption conditions, filing deadlines, and cross-act references.

**Baseline retrieval metrics (naive chunks + semantic search only):**

| Metric | Score |
| --- | --- |
| Faithfulness | 0.9360 |
| Context Precision | 0.2305 |
| Context Recall | 0.149 |
| Answer Correctness | 0.258 |

The numbers confirmed what manual inspection already showed: chunks were being retrieved that were *semantically adjacent* to the query but not actually answering it. The model was picking up on tax-domain vocabulary without landing on the right provision.

* * *

## Stage 4: Hybrid Search — Semantic + BM25

The first major improvement came from combining **dense vector search** with **BM25 lexical matching**.

### Why BM25 matters for legal text

Embedding models are excellent at capturing semantic meaning — "what is the GST rate on this type of service" maps well to the right embedding neighbourhood. But legal text is full of terms that need exact matching: section numbers (`Section 80C`), notification references (`Circular No. 183/15/2022`), HSN codes, specific defined terms. An embedding model might correctly associate `80C` with deductions but miss the specific sub-section being asked about. BM25 catches these exact-match signals.

BM25 builds on TF-IDF (Term Frequency-Inverse Document Frequency) — measuring how significant a term is relative to the full corpus — and refines it with document-length normalisation and term saturation to prevent common words from dominating results.

### Reciprocal Rank Fusion (RRF)

Results from semantic search and BM25 are merged using **Reciprocal Rank Fusion**. RRF assigns each chunk a score based on its rank in each list (not its raw similarity score), then sums these across both lists. This avoids the score-scale mismatch between cosine similarity and BM25 scores, producing a stable combined ranking.

**Metrics after hybrid search + RRF:**

| Metric | Baseline | Hybrid Search |
| --- | --- | --- |
| Faithfulness | 0.9360 | 0.9234 |
| Context Precision | 0.2305 | 0.3510 |
| Context Recall | 0.149 | 0.213 |
| Answer Correctness | 0.258 | 0.3111 |

* * *

## Stage 5: Reranking

After fusion, the top-K candidates are re-scored by a **cross-encoder reranker**. Unlike bi-encoder embeddings (which encode query and document independently), a cross-encoder takes the (query, chunk) pair jointly and produces a relevance score with full attention across both.

This is significantly more compute-intensive — you cannot precompute cross-encoder scores at index time — but since it only runs on the top-K candidates (typically 20–50), the latency overhead is manageable.

For a tax domain, reranking provides a meaningful signal: a chunk about GST input tax credit and a chunk about income tax deductions may embed similarly against a generic "tax credit" query, but a cross-encoder can distinguish the domain mismatch.

**Metrics after hybrid search + reranking:**

| Metric | Baseline | Hybrid | Hybrid + Rerank |
| --- | --- | --- | --- |
| Faithfulness | 0.8560 | 0.9234 | 0.9364 |
| Context Precision | 0.2305 | 0.3510 | 0.4108 |
| Context Recall | 0.149 | 0.213 | 0.4494 |
| Answer Correctness | 0.258 | 0.3111 | 0.4566 |

* * *

## Stage 6: Contextual Chunking — The Game Changer

Even with hybrid search and reranking, a root cause remained unaddressed: **chunks were stripped of the context they needed to be interpretable**.

Consider a chunk extracted from the GST Act:

> *"The registered person shall not be allowed to take input tax credit in respect of any supply of goods or services or both after the due date..."*

This is a perfectly valid sentence. But without knowing which section it comes from, which financial year's amendment applies, or what "registered person" refers to in context — it's ambiguous. A retrieval system can fetch it, but an LLM generating an answer from it may fill in the wrong context.

### The Contextual Chunking Approach

Contextual chunking, introduced by Anthropic, prepends a short LLM-generated context summary to each chunk *before* embedding it and building the BM25 index. The context is generated by passing the full source document alongside the target chunk to the model with this prompt structure:

```plaintext
<document>
{{Context}}
</document>
 
<chunk>
{{CHUNK_CONTENT}}
</chunk>
 
Give a short, succinct context (50–100 tokens) that situates this chunk within 
the overall document to improve search retrieval. Respond only with the context.
```

The output for the chunk above might become:

> *"This chunk is from Section 16(4) of the CGST Act 2017, which sets the time limit for claiming input tax credit by a registered taxpayer. The company's revenue grew by 3% over the previous quarter."*

This 50–100 token prefix — added to both the embedding and the BM25 index — dramatically increases the chunk's specificity. Retrieval now benefits from the document-level signal without requiring the entire document to be embedded.

> **Note on cost:** Generating context for every chunk at indexing time does incur LLM API costs. For large corpora, prompt caching (available on the Claude API) can reduce this significantly — up to 90% cost reduction on repeated document prefixes.

### Why Other Approaches Underperform

Generic document summaries appended to chunks provide marginal gains — a summary of the entire GST Act tells you very little about which specific provision a chunk belongs to. Hypothetical Document Embedding (HyDE) and summary-based indexing have also been benchmarked and show lower performance than chunk-specific contextualisation for retrieval tasks.

**Metrics after contextual chunking:**

| Metric | Baseline | Hybrid + Rerank | Contextual + Hybrid + Rerank |
| --- | --- | --- | --- |
| Faithfulness | 0.8560 | 0.9234 | 0.9364 | 0.9256|
| Context Precision | 0.2305 | 0.3510 | 0.4108 |0.813|
| Context Recall | 0.149 | 0.213 | 0.4494 |0.8723 |
| Answer Correctness | 0.258 | 0.3111 | 0.4566 |0.7957|

Anthropic's benchmarks on their datasets show contextual retrieval reducing failed retrievals by **49%** over standard embedding-only RAG, and by **67%** when combined with reranking. Results in domain-specific corpora like legal/tax text tend to be on the higher end of this range due to how heavily context-dependent the language is.

* * *

## Full Pipeline Architecture

```plaintext
Raw Documents (GST Act, IT Act, Circulars, Notifications)
         │
         ▼
   Metadata Extraction
   (source, section, effective_date, doc_type)
         │
         ▼
   Recursive Text Chunking (~400-500 tokens)
         │
         ▼
   Contextual Prefix Generation (LLM, per chunk)
         │
   ┌─────────────────┐
   ▼                          ▼
  Dense Embedding         BM25 Index
  (bge-m3, 1024d)     (contextual chunks)
  → Qdrant                 │
     │                      │
     └──────┬───────┘
                   ▼
         Reciprocal Rank Fusion
                  │
                  ▼
           Cross-Encoder Reranker
                  │
                  ▼
            Top-K Chunks → LLM → Answer
```

* * *

## Key Takeaways

**Overlap alone doesn't solve context loss.** For technical/legal text where document structure carries meaning, naive overlap is insufficient. Context needs to be explicitly injected.

**BM25 is not obsolete.** Lexical matching remains essential for identifier-heavy domains — section numbers, notification codes, HSN codes. Pure semantic search misses these.

**Reranking is the cheapest meaningful improvement.** Cross-encoder reranking on top-K candidates adds latency (100–300ms typically) but no indexing cost, and the quality gain is significant.

**Contextual chunking compounds with everything else.** It improves both the embedding quality and the BM25 index simultaneously, so every downstream step benefits.

**Evaluation set quality is everything.** 100 hand-labelled Q&A pairs covering the actual distribution of user queries is worth more than automated generation at 10x the size. Garbage evals produce misleading metrics.

* * *

## References

*   [Contextual Retrieval — Anthropic Engineering Blog](https://www.anthropic.com/engineering/contextual-retrieval)
    
*   [BAAI/bge-m3 — FlagEmbedding](https://huggingface.co/BAAI/bge-m3)
    
*   [RAGAS — RAG Evaluation Framework](https://docs.ragas.io)
    
*   [Qdrant Vector Database](https://qdrant.tech/documentation/)
    
*   [Reciprocal Rank Fusion (Cormack et al., 2009)](https://dl.acm.org/doi/10.1145/1571941.1572114)