Introduction: A Viral AI Showdown
Artificial intelligence benchmarks have long been a contentious topic—but when Pokémon enters the picture, the debate takes an unexpected turn.
Last week, a post on X (formerly Twitter) set social media ablaze. The claim? Google’s Gemini model had outperformed Claude, Anthropic’s flagship language model, in the original Pokémon video game trilogy. According to the viral tweet, Gemini had reached Lavender Town, while Claude remained stuck at Mount Moon as of late February.
“Gemini is literally ahead of Claude atm in Pokémon after reaching Lavender Town.”
— @Jush21e8 on X
But beneath the fun and fanfare lies a more serious issue: benchmarking inconsistency.
Gemini is literally ahead of Claude atm in pokemon after reaching Lavender Town
— Jush (@Jush21e8) April 10, 2025
119 live views only btw, incredibly underrated stream pic.twitter.com/8AvSovAI4x
Gemini vs. Claude: The Pokémon Controversy
The race between AI models in a retro video game might sound lighthearted, but it highlights a deeper truth about how we evaluate AI performance.
The Gemini model’s success was showcased on a Twitch stream, which allowed real-time observation of its progress. Meanwhile, Claude was lagging behind in a similar experiment. On the surface, it appeared Gemini had the upper hand in terms of reasoning and decision-making. But context matters.
The Advantage of a Custom Minimap
Reddit users were quick to uncover a critical detail: the Gemini stream utilized a custom-built minimap.
This custom tool enabled Gemini to recognize specific game elements—like cuttable trees—without having to analyze raw visual inputs. This dramatically reduced the cognitive load on the model and made it easier for Gemini to navigate the in-game world.
In contrast, Claude was playing the game “blind,” relying solely on screen captures to interpret and make decisions—an inherently more difficult setup.
This raises the question: Was Gemini really outperforming Claude? Or was it just benefiting from a more supportive environment?
Benchmarks: Beyond Entertainment
While Pokémon isn’t a serious AI benchmark, the situation illustrates the broader problem: benchmark manipulation through custom implementations.
The Case of SWE-bench Verified
Take, for example, the SWE-bench Verified benchmark, designed to assess a model’s coding proficiency. Anthropic reported two different scores for its Claude 3.7 Sonnet model:
- 62.3% accuracy with the default setup
- 70.3% accuracy when using a “custom scaffold”
This scaffold essentially structured the code input and output in a way that better suited the model—once again proving that how a test is implemented can significantly impact results.
Meta’s Llama 4 Maverick and LM Arena
Meta took a similar approach with Llama 4 Maverick, fine-tuning it for the LM Arena benchmark. The customized version of the model scored significantly better than the vanilla release, underscoring the same issue.
When the rules of the game can be bent—or rewritten—it becomes harder to assess true performance.
Why Benchmark Consistency Matters
Benchmarks are supposed to provide objective, apples-to-apples comparisons between models. But when each company tailors the testing environment to its model’s strengths, we lose the ability to evaluate AI fairly and transparently.
These discrepancies muddy the waters, making it nearly impossible for developers, businesses, and researchers to know which models truly excel and under what conditions.
What This Means for AI Stakeholders
For entrepreneurs, marketers, and tech decision-makers, these benchmarking discrepancies introduce risk. Selecting an AI model based on skewed results could lead to:
- Poor product performance
- Higher costs due to rework
- Missed innovation opportunities
Understanding the nuances behind these evaluations is critical before integrating AI into your tech stack or marketing tools.
Final Thoughts: Toward Better AI Evaluation
Pokémon may be a nostalgic battleground for AI models, but the underlying debate it sparks is critically important. As benchmarks become more gamified and customized, we must ask ourselves:
Are we evaluating AI—or just optimizing for test results?
The future of responsible AI development relies on standardized, transparent benchmarks that prioritize real-world applicability over leaderboard scores.
