The most valuable idea in Bing's history was a change to how ad headlines display. An engineer proposed combining an ad's title with the first line of text below it, so the headline could run longer. The program managers who reviewed it rated it a low priority, and it sat in the backlog for more than six months. Then another engineer ran it as a controlled experiment anyway. Revenue rose about 12 percent. In the United States alone, the change was worth more than 100 million dollars a year. When Ronny Kohavi and Stefan Thomke retold the story in Harvard Business Review in 2017, they called it the best revenue-generating idea Bing ever had.
The detail that should unsettle you is not the money. It is that the people whose full-time job was judging ideas had ranked this one near the bottom. Not maliciously, not carelessly - they looked at a longer ad headline and saw a trifle, the same way you or I would. Every organization I have worked with runs some version of that meeting. The idea is not bad. It just looks small, and looking small is fatal.
Kohavi's team at Microsoft put a number on how often that instinct is wrong. Their 2009 paper "Online Experimentation at Microsoft" reports it in one line: "Evaluating well-designed and executed experiments that were designed to improve a key metric, only about one-third were successful at improving the key metric." One in three. The other two did nothing, or made the metric worse. At Bing, a later review Kohavi co-authored with Alex Deng and Lukas Vermeer put the hit rate around 15 percent. These are elite teams that prune weak ideas hard before anything reaches a test, and they still miss two times out of three.
So here is my opinion: the one-third is not a humbling statistic to be quoted at conferences. It is a design spec. If most ideas will fail no matter who picks them, then the skill of marketing is not picking winners. It is making bets that are cheap enough, fast enough, and attributable enough that the two-out-of-three failures cost a week each instead of a quarter each.
The economics are not subtle. An untested idea that fails costs its full price: the build, the launch, the media spend, and then months of confused analytics while everyone argues about what went wrong. A tested idea that fails costs a few days and one clean read. Six months of certainty at Bing cost more than the test ever would. The experiment settled, in days, what half a year of confident opinion could not.
The taste objection
In March 2009, Doug Bowman, Google's first visual designer, resigned and published a farewell post titled "Goodbye, Google." His complaint: a company that "couldn't decide between two blues" and was "testing 41 shades between each blue to see which one performs better," where he had to prove his case on whether a border should be three, four, or five pixels wide. "I can't operate in an environment like that," he wrote, and left for Twitter. The post became the canonical case against testing culture.
He is half right, and I want to give him the half he earned. Grinding every pixel decision through an experiment pipeline is a stupid way to use a designer. But look at what the one-third says about the alternative. Without the test, the decision does not go to taste. It goes to whoever is most confident in the room, and the most confident person in the room is wrong two times out of three. The discipline is not "test everything." It is "never spend real money on an untested opinion." Border widths are free opinions. Budgets are not.
A month on the board
This is why I like how the leaderboard here is built. The minimum bid is five dollars. The board resets on the 1st of every month, and your rank is your total bid value for that month, ties broken by the most recent payment. Read that again as an experiment harness: a small floor, a fixed window, a visible scoreboard. A month in a given position tells you something no meeting can - whether that position, at that price, for your site, actually sends the people who matter.
The creative half of the bet is cheap to vary, too. Every paid spot ships with an auto-made promo video, 45 to 60 seconds, built from the listing you submit. If this month's pitch underperforms, next month's listing is a new bet at the same low floor. The failure that would have burned a quarterly campaign burns thirty days instead.
None of this makes your ideas better. It makes your wrong ideas cheaper, and that is the part you control. Take the idea you are surest about, give it its cheapest honest test before the 1st, and let the month's numbers do the arguing.