RAG is not a product: where LLM features actually earn their keep

The request arrives in roughly this form: "We want AI in the product. Can you build a chatbot over our documentation?"

We have built that. It got 12% adoption in month one, 4% in month three, and the questions it answered confidently and wrongly were exactly the high-stakes ones — pricing, eligibility, refund policy — because those are the questions whose answers live in the gaps between documents.

Retrieval-augmented generation is a technique. It is a good technique. It is not a product, and "we added RAG" is not a value proposition. What follows is the pattern we have extracted from nine shipped LLM features and five that we killed during evaluation, which is the more useful number.

The four shapes that survive contact with users

1. Structured extraction from unstructured input

This is the highest-ROI LLM application in enterprise software and it is deeply unglamorous. Someone receives a purchase order as a PDF, or an invoice photographed on a phone, or an order placed as a WhatsApp message in Hinglish, and a human retypes it into a system. The LLM reads the mess and emits a schema.

Why it works:

• The output is structured and therefore verifiable. You can validate it against a JSON schema, cross-check the arithmetic, and confirm the GSTIN exists.
• There is a ground truth, so you can measure field-level accuracy honestly.
• There is an existing cost baseline — the human doing it today — so the ROI is arithmetic, not a vibe.
• The failure mode is bounded: a low-confidence extraction goes to a human queue instead of into the database.

The critical implementation detail is that the model must never be the last line of defence:

$result = $llm->extract(
    document: $pdfText,
    schema: InvoiceSchema::json(),   // strict JSON schema, enforced by the API
);

$invoice = InvoiceDTO::fromArray($result);

$checks = [
    'line_items_sum'  => $invoice->lineTotal() === $invoice->subtotal,
    'gst_arithmetic'  => $invoice->taxIsConsistent(),
    'gstin_format'    => Gstin::isValid($invoice->supplierGstin),
    'supplier_known'  => Supplier::where('gstin', $invoice->supplierGstin)->exists(),
    'date_plausible'  => $invoice->date->between(now()->subYear(), now()->addWeek()),
];

if (in_array(false, $checks, true)) {
    return ReviewQueue::push($invoice, failed: array_keys($checks, false, true));
}

$invoice->persist();

Note what that code is doing: the model proposes, deterministic code disposes. On the invoice pipeline we shipped, 83% of documents pass every check and go straight through. 17% go to a human, who now reviews rather than retypes. That is roughly 90 hours a month back, and it is measurable in the payroll.

2. Classification and routing

Support tickets to teams. Leads to sales tiers. Uploaded documents to categories. Free-text complaints to a taxonomy.

These work for the same reason: a fixed, small output space you can evaluate with a confusion matrix. Crucially, compare against a real baseline before you build. On one project, a logistic regression over TF-IDF hit 89% on ticket routing. The LLM hit 93%. The LLM cost 200x more per call and added 800 ms of latency. We shipped the logistic regression and used the LLM only for the residual "unclear" bucket. Nobody put that in a press release, and it was the right call.

3. Drafting inside an existing human workflow

The agent still writes the reply — but they start from a draft instead of a blank box. The model's output is always reviewed by a human who was going to be in the loop anyway, so the marginal cost of a bad output is that the human deletes it.

This is where a large fraction of real LLM value sits, and it is unsexy precisely because it does not remove the human. Measure it as time-to-first-response and edit distance from draft to sent — if agents are rewriting 80% of the draft, the feature is a net negative and you should kill it. We have killed one on exactly that metric.

4. Search that returns citations, not prose

This is the one thing in the neighbourhood of the doc-chatbot idea that actually works. The difference is decisive: do not synthesise an answer. Find the passage and show it, with a link, in context. The model's job is retrieval and ranking, not authorship. It cannot hallucinate a policy that does not exist if it is not allowed to write prose.

Users trust it more, because they can verify it. And you can evaluate it properly — recall@k against a labelled set — instead of arguing about whether an answer "felt right".

Build the eval set before you build the feature

This is the single practice that separates the LLM features that shipped from the ones we killed, and it is the practice teams skip.

Before writing any application code, we collect 150 to 300 real, labelled examples — actual documents, actual tickets, actual queries — with the correct answer written by someone who knows the domain. Then we build the dumbest possible baseline and measure it. Then we build the real thing and measure it against the same set, in CI, on every prompt change.

The reason this matters is that prompt changes are code changes with no type system. Without an eval set, "improving" the prompt is guessing, and you will regress a case you fixed last month without ever knowing. We have watched a two-word prompt tweak drop extraction accuracy on one field from 96% to 71%. The eval caught it in eleven seconds. Manual testing would not have caught it at all.

If you cannot describe how you will measure whether the feature works, you are not ready to build it. This is true of all software and it is fatal with LLMs, because the output is plausible even when it is wrong.

On RAG specifically

When retrieval genuinely is the right architecture, the lesson we keep relearning is that the retrieval is the product and the generation is a thin layer on top. Almost every "the AI gave a wrong answer" bug is a retrieval bug: the right chunk was never in the context.

What has actually moved our numbers, in order:

1. Hybrid search. Pure vector search is bad at exact terms — part numbers, error codes, policy clause references, proper nouns. BM25 plus vectors, fused with reciprocal rank fusion, beats either alone by a wide margin. This is the biggest single win and it is not a machine learning technique, it is Lucene.
2. A reranker over the top 50. Retrieve wide and cheap, rerank narrow and expensive. Consistently worth its latency.
3. Chunking that respects document structure. Split on headings and semantic units, not on a 512-token sliding window. Carry the section heading into the chunk text so an isolated paragraph still knows what it is about.
4. Metadata filters. Half of "the AI is wrong" turns out to be the model quoting a document that was superseded in 2023. Filter on effective date, region, product line before the semantic search, not after.

And on infrastructure: if your data already lives in Postgres, use pgvector. A separate vector database is a second system to operate, back up, secure and keep in sync, and for corpora under a few million chunks it buys you nothing. We have removed a dedicated vector store from a client's stack and their p95 got faster, because the metadata filter and the vector search were now the same query instead of two round trips plus a join in application code.

The Indian context: DPDP is not a footnote

If your prompt contains a customer's name, phone number, address or transaction history, you are transferring personal data to a processor, and under the DPDP Act that has consequences. Concretely, on every LLM feature we ship for an Indian client:

• Redact before the boundary. A deterministic PII scrubber runs on the prompt before it leaves your infrastructure. Names become tokens, phone numbers become tokens, and you map them back on the way out. Most tasks do not need the real name to work.
• Know where inference happens and have it in the DPA. "Our vendor's vendor" is not an answer a Data Protection Board will accept.
• Zero-retention endpoints. Every major provider offers them. Use them, and get it in writing.
• Log the prompt and the response for auditability — and then treat that log as the personal-data store it now is, with the same retention and access controls as your database.
• Purpose limitation. Data collected to fulfil an order cannot be silently repurposed to train or tune a model. Consent is specific.

The test we apply before starting

1. What is the task, stated so precisely that two people would grade an output identically?
2. What does it cost today, in hours or rupees?
3. What is the non-LLM baseline, and what does it score?
4. What is the accuracy threshold below which this is worse than useless?
5. What happens when it is wrong, and who catches it?
6. What is the per-request cost and latency at 100x today's volume?

Five of our fourteen candidate features died at question 3 or 4. That is not a failure of the process; that is the process working. The cheapest LLM feature is the one you evaluate honestly and decide not to build.