This week, Sakana AI, a Nvidia-backed startup that’s raised hundreds of millions of dollars from VC firms, made a remarkable claim. The company said it had created an AI system, the AI CUDA Engineer, that could effectively speed up the training of certain AI models by a factor of up to 100x.

The only problem is, the system didn’t work.

Users on X quickly discovered that Sakana’s system actually resulted in worse-than-average model training performance. According to one user, Sakana’s AI resulted in a 3x slowdown — not a speedup.

What went wrong? A bug in the code, according to a post by Lucas Beyer, a member of the technical staff at OpenAI.

“Their orig code is wrong in [a] subtle way,” Beyer wrote on X. “The fact they run benchmarking TWICE with wildly different results should make them stop and think.”

In a postmortem published Friday, Sakana admitted that the system has found a way to — as Sakana described it — “cheat” and blamed the systems’s tendency to “reward hack” — i.e. identify flaws to achieve high metrics without accomplishing the desired goal (speeding up model training). Similar phenomena has been observed in AI that’s trained to play games of chess.

According to Sakana, the system found exploits in the evaluation code that the company was using that allowed it to bypass validations for accuracy, among other checks. Sakana says it has addressed the issue, and that it intends to revise its claims in updated materials.

“We have since made the evaluation and runtime profiling harness more robust to eliminate many of such [sic] loopholes,” the company wrote in an X post. “We are in the process of revising our paper, and our results, to reflect and discuss the effects […] We deeply apologize for our oversight to our readers. We will provide a revision of this work soon, and discuss our learnings.”

Props to Sakana for owning up to the mistake. But the episode is a good reminder that if a claim sounds too good to be true, especially in AI, it probably is.