Let’s talk about the elephant in the room: benchmarking AI ‘reasoning’ models is turning into a luxury most of us can only dream about. 🚀 Sure, big names like OpenAI and Anthropic are pushing boundaries, especially in tricky areas like physics. But here’s the catch—proving these models actually work is costing an arm and a leg. Artificial Analysis dropped a bombshell: testing OpenAI’s o1 reasoning model across seven benchmarks sets you back $2,767.05. Yeah, you read that right. That’s more than some people’s rent.
So, why’s it so expensive? Blame the tokens. These models spit out millions, sometimes tens of millions, of tokens during evaluation. And since AI companies love charging by the token (because why not?), the bills skyrocket faster than you can say ‘Elon Musk’s next big idea.’ Take OpenAI’s o1, for example—it generated over 44 million tokens in tests. That’s eight times more than GPT-4o. 💸
But hey, it’s not all bad news. The industry’s catching on. Artificial Analysis, for one, is beefing up its benchmarking budget to keep up with the flood of reasoning models. And while some models, like OpenAI’s o1-mini, are easier on the wallet ($141.22), the message is loud and clear: reasoning models are the Ferraris of benchmarking—expensive to take for a spin.
This brings us to a sticky question: as benchmarking starts to feel like a rich man’s game, how do we keep things fair and square? With labs offering freebies or discounts for testing, it’s getting harder to tell who’s independent and who’s just playing for the team. Ross Taylor from General Reasoning hit the nail on the head: ‘If you publish a result that no one else can replicate with the same model, is it even science, or just fancy marketing?’
The takeaway? The AI revolution is charging ahead, but the price tag is getting steeper. As we dive deeper into this brave new world, finding the sweet spot between cutting-edge innovation and keeping things open and affordable is crucial. Because, let’s face it, what’s the use of creating super-smart AI if testing it requires winning the lottery first?
