Multiply that across a backlog of 50 "good first issue" tickets and you're looking at entire engineering-days spent on work that's often mechanical: locate → understand → patch → verify → submit.

I wanted to see how much of that loop could be automated — not with a single LLM call and a prayer, but with a system that actually mirrors how an experienced engineer works through an issue.

💡 Prefer hands-on testing? Scroll to the bottom of this post for an interactive live demo and repository links!

What I Built

resolvo takes two inputs — a GitHub issue and a repository URL — and returns a pull request with passing tests. No local clone. No manual setup. Everything runs remotely, and you get an SSE-streamed view of progress in real time.

The measured result: turnaround time on lightweight-to-medium issues drops by roughly 85%.

Core Features

Zero local footprint — the entire fix-and-verify loop happens in a remote sandbox. Your machine never touches the repo.

Smart triage — not every issue needs the same amount of machinery. The system classifies each issue by complexity and routes it down a lighter or heavier path accordingly, so simple fixes don't pay the cost of an exhaustive analysis.

Real understanding, not pattern-matching — before writing a single line of code, the system builds an actual map of the codebase: which functions call which, which files depend on which, and where the relevant logic actually lives. This is what separates a fix that compiles from a fix that's correct.

Tests are generated and run, not assumed — every fix is verified in an isolated sandbox before it ever reaches a PR. If tests fail, the system iterates on its own fix.

Built-in code review — a review pass checks the diff for quality issues before the PR goes out, and can request changes just like a human reviewer would, triggering another round of fixes.

Human-in-the-loop where it matters — for anything flagged as high-risk or high-complexity, the system stops and hands off for human review rather than merging blind.

Live progress, not a black box — you see each stage as it happens, not just a final "done."

Architecture Decisions Worth Highlighting

1. Not every issue deserves the same treatment

One of the earliest lessons: treating a one-line typo fix and a cross-module refactor with the same pipeline is wasteful and slow. The system makes an early, fast classification pass and picks one of three paths — a fast track for trivial changes, a standard path for typical fixes with full test coverage, and a critical path for anything that always requires a human to sign off before merging. This routing decision alone is a major contributor to the speed gains on lightweight and medium-weight issues — they simply skip stages that heavier issues need.

2. Understanding before editing

Rather than asking a model to guess at a fix from a raw diff view, the system first constructs a structural understanding of the target repository — how files relate to each other, what depends on what. This map is what the planning stage draws from when deciding exactly which files need to change and why, before any code is written.

3. Multiple signals for finding the right code

Locating the actual file(s) relevant to a fix is often the hardest part of debugging — harder than writing the fix itself. Rather than relying on a single retrieval strategy, the planning stage combines several independent signals (keyword-based search, semantic re-ranking, symbol name matching, and dependency-aware expansion) and merges them into a single ranked shortlist. Diversity of signal beats depth of any one signal.

4. Verification is not optional

A generated diff isn't trusted until it's been tested in a live, isolated environment. This closes the loop that a lot of "AI writes code" demos skip — a fix that looks plausible but doesn't actually pass tests is worse than no fix at all.

5. Review as a built-in gate, not an afterthought

Before a PR reaches a human, it goes through an automated review pass that can reject the change and send it back for another iteration — with the feedback carried forward so the next attempt actually addresses what was flagged, rather than starting from scratch.

Why This Matters for Lightweight and Medium Issues

The 85% turnaround improvement isn't evenly distributed — it's heavily concentrated in the issue categories that make up the bulk of most backlogs:

Lightweight issues (typos, small logic bugs, minor config changes) skip the heaviest analysis stages entirely and move through a fast, lightly-reviewed path — these go from "sit in the backlog for days" to "resolved in minutes."

Medium issues (a bug that touches 2-3 files, a small feature with defined scope) benefit most from the automated retrieval-and-planning stage — the part of the work that normally costs a human the most thinking time before they write a single line of code.

Critical/complex issues still get a human in the loop by design — this system isn't trying to replace judgment on high-stakes changes, just remove the grunt work leading up to that judgment call.

The net effect: engineers spend their time on the 10-20% of issues that genuinely need human judgment, while the mechanical majority resolve themselves.

What's Next

I'm continuing to refine the classification thresholds and expanding language support beyond the current scope. If there's interest, I'll follow up with a deeper post on the evaluation methodology — how "success" is measured across issue types, and where the system still needs a human's help.

Have you tried automating parts of your issue-resolution workflow? I'd be curious to hear what's worked (or hasn't) for your team.

🚀 Live Demo & Repository

Ready to see how resolvo converts a GitHub issue into a tested PR in real time?

• 👉 Launch the Interactive Live Demo (No local setup required)

• ⭐ Check out the Code on GitHub

Indian tax law — GST, Income Tax, circulars, notifications — is dense, cross-referential, and unforgiving of ambiguity. A chatbot that retrieves the wrong section confidently is worse than one that admits it doesn't know. Getting retrieval right is not optional; it's the whole game.

This post documents the full journey of building a production RAG pipeline for a tax Q&A system: data collection, chunking, vector ingestion, hybrid search, reranking, and finally the technique that made the biggest difference — contextual chunking.

Stage 1: Data Collection and Corpus Preparation

The first step was sourcing and cleaning the raw corpus: GST acts, Income Tax Act sections, CBDT/CBIC circulars, notifications, and FAQs. Each document was parsed and enriched with structured metadata before storage:

• source — act name, circular number, or notification reference

• section / chapter — structural position within the document

• effective_date — relevant for time-bound provisions

• document_type — act | circular | notification | FAQ
  Metadata isn't decoration — it becomes a filter layer during retrieval, letting you scope searches to a specific document type or date range.

Stage 2: Chunking Strategy (v1 — Naive Overlap)

Standard recursive character splitting with overlap. Chunks of ~400–500 tokens with a 50-token overlap, metadata injected into each chunk's payload.

The intuition behind overlap is sound: if a concept spans a chunk boundary, the overlap ensures neither chunk loses it entirely. In practice, for dense legal text, this assumption breaks. A 50-token overlap cannot carry the subject of a section header that appeared 300 tokens ago.

Ingested into a Qdrant vector store using BAAI/bge-m3 embeddings (dense-only, 1024 dimensions).

Stage 3: Baseline Evaluation

A hand-labelled evaluation set of 100 question–answer pairs was prepared, covering a representative spread of GST and Income Tax queries — section lookups, rate queries, exemption conditions, filing deadlines, and cross-act references.

Baseline retrieval metrics (naive chunks + semantic search only):

The numbers confirmed what manual inspection already showed: chunks were being retrieved that were semantically adjacent to the query but not actually answering it. The model was picking up on tax-domain vocabulary without landing on the right provision.

Stage 4: Hybrid Search — Semantic + BM25

The first major improvement came from combining dense vector search with BM25 lexical matching.

Why BM25 matters for legal text

Embedding models are excellent at capturing semantic meaning — "what is the GST rate on this type of service" maps well to the right embedding neighbourhood. But legal text is full of terms that need exact matching: section numbers (Section 80C), notification references (Circular No. 183/15/2022), HSN codes, specific defined terms. An embedding model might correctly associate 80C with deductions but miss the specific sub-section being asked about. BM25 catches these exact-match signals.

BM25 builds on TF-IDF (Term Frequency-Inverse Document Frequency) — measuring how significant a term is relative to the full corpus — and refines it with document-length normalisation and term saturation to prevent common words from dominating results.

Reciprocal Rank Fusion (RRF)

Results from semantic search and BM25 are merged using Reciprocal Rank Fusion. RRF assigns each chunk a score based on its rank in each list (not its raw similarity score), then sums these across both lists. This avoids the score-scale mismatch between cosine similarity and BM25 scores, producing a stable combined ranking.

Metrics after hybrid search + RRF:

Stage 5: Reranking

After fusion, the top-K candidates are re-scored by a cross-encoder reranker. Unlike bi-encoder embeddings (which encode query and document independently), a cross-encoder takes the (query, chunk) pair jointly and produces a relevance score with full attention across both.

This is significantly more compute-intensive — you cannot precompute cross-encoder scores at index time — but since it only runs on the top-K candidates (typically 20–50), the latency overhead is manageable.

For a tax domain, reranking provides a meaningful signal: a chunk about GST input tax credit and a chunk about income tax deductions may embed similarly against a generic "tax credit" query, but a cross-encoder can distinguish the domain mismatch.

Metrics after hybrid search + reranking:

Stage 6: Contextual Chunking — The Game Changer

Even with hybrid search and reranking, a root cause remained unaddressed: chunks were stripped of the context they needed to be interpretable.

Consider a chunk extracted from the GST Act:

"The registered person shall not be allowed to take input tax credit in respect of any supply of goods or services or both after the due date..."

This is a perfectly valid sentence. But without knowing which section it comes from, which financial year's amendment applies, or what "registered person" refers to in context — it's ambiguous. A retrieval system can fetch it, but an LLM generating an answer from it may fill in the wrong context.

The Contextual Chunking Approach

Contextual chunking, introduced by Anthropic, prepends a short LLM-generated context summary to each chunk before embedding it and building the BM25 index. The context is generated by passing the full source document alongside the target chunk to the model with this prompt structure:

<document>
{{Context}}
</document>
 
<chunk>
{{CHUNK_CONTENT}}
</chunk>
 
Give a short, succinct context (50–100 tokens) that situates this chunk within 
the overall document to improve search retrieval. Respond only with the context.

The output for the chunk above might become:

"This chunk is from Section 16(4) of the CGST Act 2017, which sets the time limit for claiming input tax credit by a registered taxpayer. The company's revenue grew by 3% over the previous quarter."

This 50–100 token prefix — added to both the embedding and the BM25 index — dramatically increases the chunk's specificity. Retrieval now benefits from the document-level signal without requiring the entire document to be embedded.

Note on cost: Generating context for every chunk at indexing time does incur LLM API costs. For large corpora, prompt caching (available on the Claude API) can reduce this significantly — up to 90% cost reduction on repeated document prefixes.

Why Other Approaches Underperform

Generic document summaries appended to chunks provide marginal gains — a summary of the entire GST Act tells you very little about which specific provision a chunk belongs to. Hypothetical Document Embedding (HyDE) and summary-based indexing have also been benchmarked and show lower performance than chunk-specific contextualisation for retrieval tasks.

Metrics after contextual chunking:

Anthropic's benchmarks on their datasets show contextual retrieval reducing failed retrievals by 49% over standard embedding-only RAG, and by 67% when combined with reranking. Results in domain-specific corpora like legal/tax text tend to be on the higher end of this range due to how heavily context-dependent the language is.

Full Pipeline Architecture

Raw Documents (GST Act, IT Act, Circulars, Notifications)
         │
         ▼
   Metadata Extraction
   (source, section, effective_date, doc_type)
         │
         ▼
   Recursive Text Chunking (~400-500 tokens)
         │
         ▼
   Contextual Prefix Generation (LLM, per chunk)
         │
   ┌─────────────────┐
   ▼                          ▼
  Dense Embedding         BM25 Index
  (bge-m3, 1024d)     (contextual chunks)
  → Qdrant                 │
     │                      │
     └──────┬───────┘
                   ▼
         Reciprocal Rank Fusion
                  │
                  ▼
           Cross-Encoder Reranker
                  │
                  ▼
            Top-K Chunks → LLM → Answer

Key Takeaways

Overlap alone doesn't solve context loss. For technical/legal text where document structure carries meaning, naive overlap is insufficient. Context needs to be explicitly injected.

BM25 is not obsolete. Lexical matching remains essential for identifier-heavy domains — section numbers, notification codes, HSN codes. Pure semantic search misses these.

Reranking is the cheapest meaningful improvement. Cross-encoder reranking on top-K candidates adds latency (100–300ms typically) but no indexing cost, and the quality gain is significant.

Contextual chunking compounds with everything else. It improves both the embedding quality and the BM25 index simultaneously, so every downstream step benefits.

Evaluation set quality is everything. 100 hand-labelled Q&A pairs covering the actual distribution of user queries is worth more than automated generation at 10x the size. Garbage evals produce misleading metrics.

References

• Contextual Retrieval — Anthropic Engineering Blog

• BAAI/bge-m3 — FlagEmbedding

• RAGAS — RAG Evaluation Framework

• Qdrant Vector Database

• Reciprocal Rank Fusion (Cormack et al., 2009)

Lately, I’ve been diving deep into how Large Language Models (LLMs) actually work under the hood. Like most people, I used to treat AI like a black box—you type a prompt, magic happens, and a polished answer pops out.

But this week, I dug into the mechanics behind the screen and discovered three fascinating insights (and one major training trap) that completely changed how I think about prompting and training AI. If you're a beginner like me, let's break them down together!

1. Why "Give Me the Answer Only" Actually Breaks the AI 🧠

When I’m in a rush, my natural instinct is to prompt an LLM with something like: "Just give me the final answer, skip the explanation." Turns out, this is a terrible idea for complex logic or math.

I learned that the intermediate, step-by-step reasoning steps a model outputs (often called Chain-of-Thought) aren't just there to look pretty for us humans—they are literally generated for the model itself. LLMs predict text sequentially, token by token. Each new word it writes relies heavily on the context of the words it just wrote. When you force a model to skip its thinking process and jump straight to the conclusion, you rob it of its working memory.

💡 Lesson #1: If you want accurate results for tricky problems, always let the model think out loud!

2. The Counting Blind Spot: Why AI Fails at Basic Spelling & Counting 🔢

Have you ever asked an AI to count how many times a specific letter appears in a long word, only for it to confidently give you the wrong number? I always found this completely baffling. It's a supercomputer, right? Why can't it count to 4?

Here is the secret: AI does not see raw text character-by-character. Instead, before your text even hits the AI's "brain," a preprocessing step cuts words up into semantic chunks called Tokens.

Because the model only processes these pre-packaged token IDs, it doesn't intuitively "see" the individual letters inside them. It’s like trying to count the syllables in a word without being allowed to look at the alphabet.

The Fix: Execution Over Prediction

This is why using tools changes everything. When you tell an LLM to "use code" (like an integrated Python interpreter) to solve a problem, it stops guessing the next word based on mathematical probability. Instead, it generates a literal, deterministic script and executes it.

• Prediction: "I guess the word strawberry has 2 'r's based on common speech patterns." ❌

• Execution: print("strawberry".count("r")) -> 3 ✅

3. Training the Unquantifiable: How We Teach AI to Tell Jokes 🎭

How do you train an AI to do something completely subjective, like writing a funny joke, maintaining a helpful tone, or summarizing an essay well? There is no absolute mathematical "right answer" to check against a key.

I looked into how engineers solve this at scale, and it comes down to an awesome process called RLHF (Reinforcement Learning from Human Feedback):

1. Human Scoring: Humans are given multiple variations of an AI response to a single prompt and rank them from best to worst.

2. The Reward Model: That ranking data is fed into a separate "referee" neural network to teach it what a "good" human response looks like.

3. The Loop: The main AI generates text, the referee network scores it, and the main AI adjusts its internal parameters to chase higher scores.

4. The Over-Training Trap 🛑

You would think that leaving a model in this reinforcement loop longer would make it smarter and smarter, right? This was my absolute favorite finding this week: it doesn't!

I learned about a fascinating concept where response quality behaves like an inverted U-curve relative to training time.

If you let the training loop run too long without intervention, the quality drops off a cliff. The AI starts "gaming the system." It figures out exactly what quirks or phrases the referee network scores highly, and it begins outputting overly long, repetitive, or incredibly sycophantic ("brown-nosed") answers. They score perfectly on paper but read horribly to a real human.

Knowing exactly when to hit the brakes on training is a literal science!

Wrapping Up 🚀

Writing this all out helped me realize that prompt engineering isn't just about finding "magic words"—it’s about understanding the underlying architecture of the machine you are collaborating with.

If you're also experimenting with AI tools, try letting them write out their reasoning next time or explicitly ask them to use a code block for calculation, and watch your results drastically improve.

What did you learn in your tech journey this week? Let me know in the comments below, and don't forget to follow along for more beginner-friendly tech roundups!