NM Company
A portfolio and enquiry site for events firm NM Company
A visual portfolio and enquiry website for an event management and supplies firm — presenting a full...
Read itLLM systems, RAG and applied ML
AI & ML Development
AI features built like engineering, not like demos — evaluation harnesses before launch, grounded retrieval with citations, hallucination guardrails and a cost-per-request budget you monitor.
- An evaluation harness in CI before a single feature ships to a customer
- Every answer grounded in a retrieved source, with a citation the user can open
- Refusal and escalation paths — the system says "I do not know" instead of inventing
- Cost per request and p95 latency budgeted, monitored and alerted on
What this is
The demo is the easy part.
The most valuable thing we do on an AI engagement is sometimes to tell you not to build one.
A regex will beat an LLM at extracting a GSTIN. A trained classifier will beat an LLM at routing a support ticket, at a thousandth of the cost and a hundredth of the latency. A SQL query will beat an LLM at "how many orders shipped late last month" — and it will be right, which the LLM will not reliably be. If the task has a deterministic answer, a deterministic system should produce it. LLMs are for language: summarising, drafting, extracting from unstructured text, conversing, and reasoning over documents where the rules cannot be written down.
When it is genuinely the right tool, we build it like engineering rather than like a demo. That means an evaluation harness before anything ships, guardrails that catch a hallucination before your customer reads it, a retrieval layer that grounds every answer in a citable source, and a cost-per-request budget you actually monitor. The demo is the easy part — anyone can produce an impressive one in a weekend. Making it correct on the four hundredth query, at a cost per query you can defend, is the work.
Nothing ships without an evaluation harness.
If you cannot measure whether your AI feature got better or worse, you are not engineering. You are gambling with a changelog.
So the first artefact on every AI project is not a prompt. It is an evaluation set: 150 to 400 real inputs from your actual domain, each with a known-good answer, assembled with your subject matter experts. It costs a week and it is the single highest-leverage week in the project.
The harness runs that set on every change — a new prompt, a new model version, a new retrieval strategy — and scores it: exact match where there is a right answer, faithfulness to the retrieved source where there is not, a rubric graded by a stronger model with human spot-checks, plus latency at p50 and p95 and cost per request. It runs in CI, exactly like a test suite, because that is what it is.
The value shows up the first time somebody says "let us just tweak the prompt". Without a harness, that is a vibe. With one, you find out in four minutes that the tweak fixed three cases and quietly broke eleven. We have watched more than one team ship a "small prompt improvement" that dropped accuracy nine points, and only discover it from customer complaints a fortnight later.
Where AI projects die
The demo worked. Then it met production.
We have been called in to fix all six of these. Five of them are avoidable in week one.
-
Nobody can say whether it is getting better
No eval set, no baseline, no gate. Every prompt change is a vibe, and a "small improvement" quietly dropped accuracy nine points a fortnight ago.
-
It invents policy with total confidence
Asked about a refund window that does not exist, it makes one up in fluent, plausible English. There is no refusal path, so refusing was never an option it had.
-
The inference bill is the new cloud bill
Twenty retrieved chunks stuffed into every prompt, the frontier model on every request including "hi", no caching. ₹3 lakh a month for something a reranker and a router would have run for ₹40,000.
-
It takes nine seconds to answer
Three sequential model calls and an unindexed vector search. Users abandon the chat and open a support ticket, which was exactly what this was meant to prevent.
-
The retrieval is bad, so the answer is bad
Documents chunked naively at 500 characters, cutting tables and clauses in half. The model is not hallucinating — it is faithfully summarising garbage.
-
Customer data went somewhere nobody checked
PII in prompts, to a consumer API tier, with no data processing agreement and no redaction. The DPDP question arrives and there is no honest answer.
The first decision
Prompt, retrieve, fine-tune — or do not use a model at all.
| Classic ML / rules | Prompt + RAG | Fine-tuning | |
|---|---|---|---|
| Answer lives in your documents | No | Best | Poor — and goes stale |
| Task is deterministic (extract, route, count) | Best | Wasteful | Wasteful |
| Needs a consistent tone or output format | No | Workable | Best |
| Cost per request | Negligible | Moderate | Low, after training |
| Answers must be citable | n/a | Yes | No |
| Updates when your data changes | Retrain | Instantly | Retrain |
| Typical time to production | 3–6 weeks | 6–12 weeks | 10–16 weeks |
What you get
The scaffolding that makes it safe to ship.
None of this is visible in the demo. All of it is what separates a feature you can defend from one you will have to apologise for.
An evaluation harness in CI
150–400 real inputs with known-good answers, scored on accuracy, faithfulness, refusal correctness, latency and cost. Runs on every prompt and model change. A regression fails the build.
Hallucination guardrails
Answers grounded strictly in retrieved context. Claims verified against source chunks. An explicit "I do not know" path with human escalation, because a system that cannot refuse will always invent.
Retrieval that actually retrieves
Semantic chunking that respects tables and clauses, hybrid keyword plus vector search, a reranker, and citations the user can open. Most "hallucination" is a retrieval bug wearing a costume.
A cost and latency budget
Difficulty routing — cheap model for the easy 80%, frontier model only where it earns it. Prompt caching, context sizing, streaming. Cost per request tracked with an anomaly alert.
A data path you can defend
PII redaction before egress, enterprise API tiers that exclude your data from training, full prompt and completion logging, and open-weight models in your own VPC where DPDP or a contract demands it.
Observability for non-determinism
Every request traced with its retrieved context, tokens, cost and latency. Thumbs-down feedback routed into the eval set, so production failures become tomorrow's regression tests.
How we build
Evaluate, then build. Never the other way round.
The order is the whole method. Teams that build first and measure later ship confidently wrong systems.
-
01
Feasibility, and the honest no
1 weekWhat is the task, what does success look like numerically, what does a wrong answer cost, and is a model even the right tool? This is where we tell you a classifier or a SQL query does the job for a hundredth of the price. Several of our best AI engagements ended here, and those clients came back.
-
02
Build the evaluation set
1–2 weeks150 to 400 real inputs from your domain, each with a known-good answer, built with your subject matter experts. Then the harness that scores against it and the thresholds that gate a release. This is the highest-leverage fortnight in the project and it happens before any feature is written.
-
03
Retrieval and data pipeline
2–4 weeksIngest, semantic chunking that respects document structure, embeddings, hybrid search, reranking. We measure retrieval quality on its own — recall at k, before generation is involved — because you cannot fix a generation problem that is actually a retrieval problem.
-
04
The system, with guardrails
3–6 weeksPrompt architecture, tool calling, structured outputs validated deterministically, citation enforcement, refusal paths and human escalation. Everything iterated against the harness, so every change is scored rather than argued about.
-
05
Cost, latency and shadow launch
1–2 weeksDifficulty routing, prompt caching, context sizing, streaming to make p95 feel faster than it is. Then we run it in shadow against real traffic with humans reviewing before anything reaches a customer, and only open it up when the numbers hold.
-
06
Production and the feedback loop
OngoingTraced requests, cost anomaly alerts, and thumbs-down feedback routed straight back into the eval set. Every production failure becomes a permanent regression test, so the same mistake cannot ship twice. Models change under you — the harness is what makes that survivable.
The budgets we hold
Numbers, not adjectives.
Every AI system we ship carries an explicit budget for correctness, speed and cost, agreed with you before build and enforced as a release gate. "It seems pretty good" is not a launch criterion we accept.
Talk about your use case
0
Eval cases in the harness
Real domain inputs, expert-labelled
0s
p95 time to first token
Streaming, with retrieval in the path
0%
Typical cost reduction
Difficulty routing, reranking, caching
0%
Answers with a citation
Or an explicit, honest refusal
What we actually build
Applications, not experiments.
Document intelligence over your own corpus
Contracts, policies, tenders, technical manuals, claim files. Grounded question answering with citations the reader can open, clause extraction into structured fields, and comparison across versions. This is where RAG genuinely earns its keep — and where naive chunking quietly ruins the whole thing by cutting a table in half.
Support assistants that escalate
Answers grounded in your help centre and order data, with a refusal path and a clean handoff to a human. Deflection measured honestly — against resolution, not against "did not open a ticket".
Classification and extraction
Ticket routing, invoice and KYC extraction, lead scoring, defect detection. Often a small trained model, not an LLM — faster, cheaper, more accurate, and we will say so.
Forecasting and recommendation
Demand forecasting, churn propensity, dynamic pricing, next-best-action. Real ML with a proper backtest, not a chatbot wearing a lab coat.
Agentic workflows, scoped tightly
Tool-calling systems that read a document, query a database, draft an action and stop for a human at the point where money or risk changes hands. Kept deliberately narrow — a broad autonomous agent is a demo, whereas a narrow one that reconciles three hundred invoices a day and flags the eleven it is unsure about is a product.
The stack
Model-agnostic on purpose.
Frontier models change every quarter. The harness, the retrieval layer and the guardrails are what persist — so we build so you can swap the model in an afternoon.
Claude API
OpenAI API
Gemini
Llama / Mistral (self-hosted)
Hugging Face
pgvector
Qdrant
LlamaIndex
LangGraph
Rerankers
Ragas
Custom eval harness
FastAPI
PyTorch
scikit-learn
Langfuse
Proof
All case studies
AI in production, with the numbers.
Systems that are measured, not demoed.
FealDeal
E-commerce — Ethnic Fashion
A high-catalogue ethnic-wear storefront for FealDeal
A Shopify storefront for a Surat ethnic-wear label — 200+ SKUs of lehengas, sarees and suits organised by type, colour, fabric and work, wit...
Shopify
Platform
0+ SKUs
Catalogue
Turant Logistics
Logistics & Supply Chain
Cutting order-to-dispatch by 62% for a 400-truck 3PL
A fleet, ePOD and route-optimisation platform built offline-first for drivers in low-connectivity corridors — because the two off-the-shelf...
0% faster
Faster order-to-dispatch on the ops floor
0% shorter
Shorter POD-to-invoice cycle (8.4 days to 19 hours)
Collection by Rehan
E-commerce — Watches & Accessories
A brand-led watch and eyewear store for Collection by Rehan
A Shopify storefront for a Surat watch and eyewear retailer — browsable by gender, brand, style and combo, with predictive search, COD check...
Shopify
Platform
Brand & style
Browse by
Whenever the task has a deterministic answer. Extracting a GSTIN or a pincode is a regex. Routing a ticket to one of nine queues is a classifier — cheaper, faster and more accurate. "How many orders shipped late last month" is a SQL query, and the LLM will occasionally get it wrong in a way that looks completely confident. Reserve LLMs for language: summarising, drafting, extracting from messy unstructured text, conversing, and reasoning over documents where the rules genuinely cannot be written down.
RAG in about nine cases out of ten. RAG changes what the model knows; fine-tuning changes how it behaves. If the answer lives in your documents, policies or database — and it usually does — retrieve it, put it in the context, and cite it. Fine-tuning is the right call for a consistent output format, a specific tone, or a narrow classification task where you have thousands of labelled examples. It is a poor and expensive way to teach a model facts, and the facts go stale the moment your documents change.
You cannot eliminate it, so you engineer around it. Ground every answer in retrieved context and refuse to answer outside it. Require citations and verify each claim maps to a retrieved chunk, dropping the ones that do not. Give the system an explicit "I do not know" path with escalation to a human — most systems hallucinate because refusal was never made available to them. Add deterministic validators for anything structured. Then measure the residual rate on your eval set and make it a release gate.
That is an architecture question and we answer it before we build, not after. We size the context — the difference between stuffing twenty retrieved chunks and reranking to the best four is often 60% of your bill. We route by difficulty, sending the easy 80% of requests to a small cheap model and escalating only what needs the frontier one. We cache aggressively. A support assistant handling 50,000 queries a month typically lands between ₹25,000 and ₹90,000 in inference cost, monitored per request.
We design the data path first. What leaves your network, to which provider, under what retention terms, and with what enterprise agreement — the API tiers that exclude your data from training are not the default and must be configured. PII is redacted before it reaches a third-party model where the use case allows. Where regulation or a customer contract requires it, we run open-weight models in your own VPC or on-premise. And we log every prompt and completion so an auditor can be answered honestly.
An evaluation set of 150 to 400 real inputs from your domain, each with a known-good answer, built with your subject matter experts in the first week. It runs in CI on every prompt, model and retrieval change, scoring accuracy, faithfulness to source, refusal correctness, p95 latency and cost per request. We agree the release thresholds with you up front. If a change drops accuracy below the gate, it does not ship — regardless of how good the demo looked.
Tell us the use case. We will tell you if it needs a model.
Sometimes the answer is a two-week classifier instead of a six-month AI programme. You will get that answer straight, and for free, in the first conversation.
Proof
Shipped, measured, still running.
Swasthya Sarathi
A healthcare-companion platform for Swasthya Sarathi
A multi-service healthcare platform that helps people find hospitals, doctors, labs, medical stores,...
Read itMV Tech Education
A course and admissions platform for MV Tech Education
A course-catalogue and admissions website for a Bihar vocational institute — leading with industrial...
Read itRelated services
Web Development
Production web applications built on Laravel, Next.js and Postgres — typed, tested, observ...
ExploreMobile App Development
Flutter, React Native and native apps that hold up offline, on old hardware and on a metro...
ExploreEcommerce Development
Commerce platforms where the checkout converts and the back office reconciles — multi-ware...
ExploreLet's talk ai & ml development
Bring us the hard version of the problem.
A senior engineer reads every enquiry. You'll get a real answer — scope, risk and a number — within one business day.