Conversion & experimentation — ecommerce and SaaS
Most of your winning tests didn't win.
I audit experimentation programmes and rebuild the ones that are producing confident answers to the wrong questions. Ten years in CRO, most of it spent finding out that the lift was noise.
Four tests a client shipped as wins — re-analysed
Three of the four intervals cross zero. The point estimates all looked positive, so all four went into the roadmap as proven. Only one of them belonged there.
What I do
Three ways teams usually bring me in.
01 — Diagnose
Programme audit
I take your last eighteen months of tests and re-run the analysis. Sample ratio mismatch, peeking, underpowered variants, metrics that never mapped to revenue. You get a list of which results you can still spend and which ones to retire.
Deliverable — audit memo + retest list02 — Design
Roadmap and test design
Hypotheses built from your own analytics rather than a swipe file. Each one comes with a minimum detectable effect, a runtime you can actually staff, and a written statement of what would make you abandon it.
Deliverable — prioritised roadmap + briefs03 — Build
In-house capability
For teams who want to stop outsourcing judgement. Analysis standards, a review ritual that catches bad calls before they ship, and enough statistics that your PMs can argue with your data team productively.
Deliverable — standards doc + trainingSelected work
Placeholder case studies — real ones go here.
Killed the checkout redesign, kept the revenue
A five-month redesign was about to ship on the strength of a test that had been stopped early three separate times. Re-analysis with a sequential correction put the true effect near zero. The team redirected the quarter into subscription retention instead.
The activation metric that wasn't
Eleven tests optimised against a proxy for activation that turned out to be uncorrelated with retention at 90 days. We rebuilt the metric tree, re-scored the backlog, and cut the roadmap by half.
Fixing a year of sample ratio mismatch
In-app browser traffic was silently failing to bucket, skewing every mobile result toward the control. Once instrumented properly, three previously flat tests turned out to be clear winners.
Experiment Notes
Short pieces on what actually goes wrong in testing.
Your exposure rate is diluting every result you have
A proxy metric is a bet that two things move together. Check the bet.
In-app browsers are quietly breaking your mobile tests
Viewport-triggered bucketing makes small wins look large
Have a test result you don't quite believe?
Send me the readout. If there's nothing wrong with it I'll tell you that too, and it won't cost you anything.
himanshu@himanshukhurana.com