Skill rating climbs 12% per win—losses teach your UX nothing

Every product team I've worked with has a story about the feature that "tested well" and then quietly failed in the wild. The numbers looked fine on the dashboard, the usability sessions were smooth, and yet six months later the thing is a ghost town. So here's the question worth sitting with: if your win rate in testing is climbing nicely, what exactly is it teaching you about the moments when users fail? Because the evidence from behavioural psychology suggests the answer might be "almost nothing" — and that has real consequences for how you build websites for Australian businesses.

The Asymmetry Nobody Budgets For

Kahneman and Tversky's work on loss aversion is the obvious starting point, but it's usually cited for consumer pricing pages when it's just as relevant to how you read your own analytics. Losses loom larger than gains. That's true for your users, and it's true for you.

Think about what happens in a typical iteration cycle. A test comes back positive — conversion up, task completion up, time-on-page up. The team celebrates, ships it, moves on. A test comes back negative — everyone gathers around the whiteboard, digs into session recordings, runs a root-cause analysis, and the whole thing gets a proper post-mortem.

Notice the imbalance. The wins get a tick. The losses get a thesis. You are systematically extracting far more insight from failure than from success, which sounds admirable until you realise it means your mental model of why things work is built almost entirely from negative evidence. You know a great deal about what breaks your interface. You know surprisingly little about what makes it sing, because you never stop to interrogate the good days.

There's a second-order problem too. When a win arrives, the temptation is to attribute it to whatever you changed. But most UX wins are confounded — seasonality, traffic mix, a competitor's outage, a change in ad creative. Without a deliberate effort to explain why a win happened, you're really just banking a number and moving on.

Reward Loops Are for Users, Not for You

Variable-ratio reinforcement — the intermittent reward schedule that B.F. Skinner documented in pigeons — is usually discussed in the context of keeping users coming back. Notifications, streaks, progress bars, the little dopamine hit of a "saved successfully" toast. Fair enough.

But here's the uncomfortable bit: your product team is on the same schedule.

Some sprints produce a clean win. Some produce nothing. Some produce a result that looks like a win until you segment by device and realise it was entirely driven by desktop traffic during a sale. The unpredictability of when the next good result arrives is exactly what keeps teams pushing — and exactly what makes them sloppy about interpretation. When a win finally lands, the relief is palpable and the scrutiny drops.

I've watched this play out with a client running a booking flow for a regional tourism operator. The team had a 12% lift in completed bookings after a form redesign and immediately shipped it to the second property in their portfolio. It tanked. The original lift turned out to be driven by a paid campaign that happened to overlap the test window, plus a competitor's site going down for maintenance. The "win" taught them nothing transferable. The loss taught them plenty — but only because they were forced to look.

The Competitive Play Problem

There's a useful analogy in competitive games and sport, where skill rating systems (Elo and its descendants) explicitly model wins and losses as different kinds of information. A win against a weaker opponent tells you very little. A loss against a stronger one tells you a lot. Good rating systems weight outcomes by the quality of the opposition, not just the result.

Your website doesn't have an opponent, but it does have context — traffic source, device, intent, time of day, whether the user arrived from a search for your brand name or a generic query. A "win" measured across a blended audience is like beating a weaker opponent: technically a result, practically uninformative.

The practical move is to stop treating aggregate conversion as your skill rating and start treating segmented performance as the thing you're actually trying to improve. If a change lifts conversion for returning desktop users but does nothing for mobile-first new visitors — which, for most Australian small business sites, is the majority of traffic — you haven't learned something to repeat. You've learned something to investigate.

Decision-Making Under Uncertainty, Applied to Your Roadmap

Most UX decisions get made under genuine uncertainty, with incomplete data and a deadline. That's fine. The problem is that teams often respond to that uncertainty by over-weighting whatever evidence is easiest to obtain — usually the most recent test result, or the loudest stakeholder's anecdote.

A more honest approach borrows from how good decision-makers handle uncertainty in other domains: they separate the decision from the outcome. You can make a well-reasoned call and still get a bad result, and vice versa. If you only review outcomes, you can't tell whether your process is any good.

Concretely, that means writing down your hypothesis before you ship, including what you expect to happen and why. Then, when the result comes in, you're comparing against a stated prediction rather than rationalising after the fact. It's a small discipline that pays off enormously over a year of iterations, because it forces you to notice when a win happened for reasons you didn't anticipate — which is precisely the situation where you stand to learn the most.

What to Do Differently Starting Next Sprint

Stop treating wins as endpoints. When something works, spend twenty minutes asking why. Segment the result. Check whether the lift holds across devices, traffic sources, and new versus returning visitors. If it doesn't, you don't have a repeatable improvement — you have a coincidence with good timing.

Keep a running log of hypotheses and outcomes, including the ones that surprised you in a good way. Review it quarterly. Patterns will emerge that no single test could have shown you.

And build in a deliberate counterweight to loss aversion: schedule a "why did this work?" review for every win, with the same rigour you'd apply to a failure. It feels indulgent. It isn't. It's the only way your understanding of your own website compounds, rather than resetting every time the numbers go your way.