Not even pokémon is safe from ai benchmarking controversy. Last Week, A Post on x Went viral, claiming that google’s latest gemini model surpassed anthropic’s flagship claude model in the original pokémon video game trilogy. Reportedly, Gemini Had Reached Lavendar Town in a developer’s twitch stream; Claude waste Stuck at Mount Moon As of late … Read more
Openai, like many ai labs, thinks ai benchmarks are broken. It says it Wants to fix them through a new program. Called the Openai Pioneers Program, The Program will focus on creating evaluations for ai models that “set the bar for what good looks like,” blog post, “As the pace of AI Adoption Accelerates Across … Read more
Thought Pokémon was a tough benchmark for aiOne Group of Researchers Argues That Super Mario Bros. is even tougher. Hao Ai Lab, A Research Org at the University of California San Diego, on Friday Threw Ai Into Live Super Mario Bros. Games. Anthropic’s Claude 3.7 Performed the Best, Followed by Claude 3.5. Google’s Gemini 1.5 … Read more
Debates over AI Benchmarks – and how they are reported by ai labs – are spilling out into public view. This week, an openai employee accused Elon Musk’s Ai Company, Xai, of Publishing Misleading Benchmark Results for its latest ai model, Grok 3. One of the co-for founders of xai, igor babushkin, Insified That the … Read more