A performance rescue: 11.2s to 640ms, nine days before renewal

Vayu Procure sells a B2B procurement portal — 900 enterprise buyers running RFQs against a network of 26,000 suppliers. The product worked. It was just too slow to use.

They called us eleven days before a renewal

Their largest customer, 19% of ARR, had written that the renewal depended on speed. The complaint was clear and fair. The supplier bid-comparison screen, the one screen a buyer uses to actually award business, was taking 11.2 seconds at P95. Their category managers had started copying data to Excel and comparing bids there. That is a sign a SaaS product is dying.

The symptoms

• Database CPU pinned at 95% through the working day, on an already oversized RDS instance they had made bigger three times.
• The nightly spend-analytics job ran for seven hours and, while it ran, made the application slow for anyone in a timezone unlucky enough to be awake.
• Their server bill had tripled in eighteen months while their user count had not quite doubled.

What they had been told to do

Two earlier consultants had suggested a rewrite in Go and splitting the database. Both would have taken months. Both would have been the wrong answer, and one of them would probably have finished after the company did.

Here is what an afternoon with a profiler and performance_schema really showed:

• An N+1 explosion. The bid list showed 200 rows and sent 4,100 queries to do it, through lazily-loaded linked data three levels deep.
• A JSON full-table scan. The line-item filter ran WHERE JSON_EXTRACT(attributes, '$.category') = ? against a 41-million-row table. No index can help that. Every filter click scanned the whole table.
• The analytics job used chunk() with an ORDER BY on an unindexed column. Every chunk sorted all 41M rows again. It was not processing data. It was re-sorting the table, 4,100 times, overnight.
• Sessions were in the database. Every single request wrote a row, even requests for static files that had no business touching the app.
• No HTTP caching at all on a supplier catalogue that changes about once a week.

Their problem was never that MySQL is slow. They were asking it the wrong question, 4,100 times a page.

The bid-comparison screen was fixed and put live in nine days. The renewal was signed. The rest of the work was released over the next six weeks.

Results

• P95 on the bid-comparison screen: 11.2s → 640ms. A 17× improvement, on the screen that decides whether their product gets used or exported to Excel.
• Database CPU at peak: 95% → 22%. Same instance, same data, same users, same MySQL. They simply stopped asking it to do silly things.
• Nightly analytics: 7 hours → 9 minutes, and running on a replica, so it can no longer slow the application for anyone in any timezone.
• Monthly infrastructure spend fell 41%. We made the RDS instance two sizes smaller, from a db.r6g.4xlarge to an xlarge. You can only do that after you no longer need the extra room. They had been paying to escape a code problem for eighteen months.

The part that outlives us

The query-budget check in CI is, honestly, the most valuable thing we left behind. Speed fixes fade over time. Without a guard, the same N+1 grows back, and someone calls another agency in eighteen months. A failing build is that guard. It has caught four problems before they went live, and it will keep catching them long after we are gone.

We spent nine days on the fix and two days on the measuring that told us which nine days to spend. The two days were the important ones. Every wrong answer this company had been given (rewrite it, split it, buy a bigger instance) came from someone who skipped them.