A performance rescue: 11.2s to 640ms, nine days before renewal

Vayu Procure sells a B2B procurement portal — 900 enterprise buyers running RFQs against a network of 26,000 suppliers. The product worked. It just could not be used.

They called us eleven days before a renewal

Their largest customer — 19% of ARR — had put the renewal in writing as conditional on performance. The complaint was specific and entirely fair: the supplier bid-comparison screen, the single screen a buyer uses to actually award business, was taking 11.2 seconds at P95. Their category managers had started exporting to Excel and comparing bids there, which is the sound of a SaaS product dying.

The symptoms

• Database CPU pinned at 95% through the working day, on an already oversized RDS instance they had upsized three times.
• The nightly spend-analytics job ran for seven hours and, while it ran, made the application slow for anyone in a timezone unlucky enough to be awake.
• Their infrastructure bill had tripled in eighteen months while their user count had not quite doubled.

What they had been told to do

Two prior consultants had recommended, respectively, a rewrite in Go and sharding the database. Both would have taken quarters. Both would have been the wrong answer, and one of them would probably have finished after the company did.

Because here is what an afternoon with a profiler and performance_schema actually showed:

• An N+1 explosion. The bid list rendered 200 rows and issued 4,100 queries to do it, through lazily-loaded polymorphic relations three levels deep.
• A JSON full-table scan. The line-item attribute filter ran WHERE JSON_EXTRACT(attributes, '$.category') = ? against a 41-million-row table. No index can help that. Every filter click was a full scan.
• The analytics job used chunk() with an ORDER BY on an unindexed column. Every chunk was a fresh filesort over 41M rows. It was not processing data; it was re-sorting the table, 4,100 times, overnight.
• Sessions were in the database. Every single request — including static asset requests that had no business touching the app — wrote a row.
• No HTTP caching at all on a supplier catalogue that changes about once a week.

Their problem was never that MySQL is slow. It was that they were asking it the wrong question, 4,100 times a page.

The bid-comparison screen was fixed and deployed in nine days. The renewal was signed. The remaining work shipped over the following six weeks.

Results

• P95 on the bid-comparison screen: 11.2s → 640ms. A 17× improvement, on the screen that decides whether their product gets used or exported to Excel.
• Database CPU at peak: 95% → 22%. Same instance, same data, same users, same MySQL. They were simply no longer asking it to do absurd things.
• Nightly analytics: 7 hours → 9 minutes, and running on a replica, so it can no longer degrade the application for anyone in any timezone.
• Monthly infrastructure spend fell 41%. We downsized the RDS instance two sizes — from a db.r6g.4xlarge to an xlarge — which is a thing you can only do after you have stopped needing the headroom. They had been buying their way out of a code problem for eighteen months.

The part that outlives us

The query-budget assertion in CI is, honestly, the most valuable artefact of this engagement. Performance rescues have a half-life: without a mechanism, the same N+1 grows back, and someone calls another agency in eighteen months. A failing build is that mechanism. It has caught four regressions before they reached production, and it will keep catching them long after we are gone.

We spent nine days on the fix and two days on the profiling that told us which nine days to spend. The two days were the important ones. Every wrong answer this company had been given — rewrite it, shard it, buy a bigger instance — came from someone who skipped them.