Pelican Bicycle Drawing: AI Models' Performance Analyzed
The performance of AI models in drawing pelicans on bicycles has been analyzed.
1Pelican Bicycle Drawing: A Benchmark Story
Simon Willison has been testing every major language model release with the prompt 'Generate an SVG of a pelican riding a bicycle' for the past few years. What started as a joke has become one of the most famous informal benchmarks in AI.
2Popularity on Hacker News
Simon's pelican bicycle drawing results are often among the most upvoted comments on Hacker News threads announcing new AI model releases. This test has now become a standard for AI labs.
3Pelicanmaxxing: Benchmark Manipulation?
With billions or even trillions of dollars at stake, a strong result could persuade users. Could AI labs be manipulating this benchmark? An experiment was conducted to answer this question.
4Experiment Details
1,008 SVGs were generated across 7 different AI models. Each image was scored by an LLM judge and analyzed using Claude Fable 5. All code is available on Github.
5Animal and Vehicle Selection
The experiment used 8 animals and 6 vehicles. The pelican and bicycle combination was chosen as a variation of Simon's famous prompt. Other animals and vehicles varied in similarity and difficulty.
6Models and Testing Process
7 different models, including GPT-5.6 Terra, Claude Sonnet 5, and Gemini 3.5 Flash, were tested. Three samples were generated for each prompt, and each image went through a three-stage pipeline.
7Visual Evaluation
Each SVG was rendered to PNG and scored by GPT-5.6 Luna for the animal, vehicle, and coherence of the action. These scores were used to compare the performance of the models.
8Pelicans' Performance
Pelicans ranked 6th out of 8 animals. If labs were working on the benchmark, pelicans would be expected to rank higher. However, this was not the case.
9Bicycles' Performance
Bicycles ranked second to last out of 6 vehicles. This suggests that labs did not specifically train on the benchmark. Drawing bicycles proved to be more challenging than other vehicles.
10Pelican Bicycle Combination
The pelican bicycle combination ranked 42nd out of 48 combinations. This result supports the idea that labs did not specifically train on the benchmark.
11Conclusion and Implications
The experiment shows that AI labs did not manipulate the pelican bicycle drawing benchmark. This increases the reliability of informal benchmarks in evaluating AI model performance.