How Do AI Models Perform on the Famous Pelican Test?
AI drawing capabilities evaluated through the iconic pelican-on-a-bicycle benchmark.
1Origin of the Pelican Test
Simon Willison has tested every major LLM release with the same prompt: 'Generate an SVG of a pelican riding a bicycle.' What began as a joke has become one of AI's most famous informal benchmarks.
2Test's Popularity
The pelican test is now so famous that there's debate about whether AI labs might be 'pelicanmaxxing' - specifically optimizing for this benchmark. Results often become top comments on Hacker News release threads.
3Research Methodology
The study created 48 prompts combining 8 animals and 6 vehicles. 1,008 SVG images were generated across 7 leading models. Each image was scored 1-5 for animal, vehicle, and scene coherence.
4Models Tested
GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2 and DeepSeek V4 Pro were compared. Three samples per prompt were generated at temperature 1.0.
5Evaluation Process
Generated SVGs were rendered to PNGs. GPT-5.6 Luna scored each image. Gemini 3.1 Flash-Lite analyzed scene elements. Only 11 regenerations were needed across all 1,008 images.
6Pelicans Underperformed
Pelican drawings ranked 6th out of 8 animals across all models. Cats, whales and raccoons were drawn better, suggesting labs aren't specifically training for pelicans.
7Bicycles Also Struggled
Bicycle drawings ranked 5th out of 6 vehicles. Simple vehicles like skateboards scored higher. 2/3 of bicycle images had missing or disconnected parts.
8Famous Combination Failed
The pelican-bicycle combination ranked 42nd out of 48. Even after difficulty adjustment, it showed no special performance, indicating no specific optimization.
9Model Comparison
Claude Sonnet 5 led overall performance, while GLM-5.2 did best on pelican-bicycle. However, its strong performance across other combinations suggested general capability rather than specific optimization.
10Future of Testing
The research shows informal benchmarks shouldn't drive model development. More comprehensive tests evaluating general capabilities are needed for proper AI assessment.
11Conclusions
While the pelican test remains a fun benchmark, there's no evidence labs are specifically optimizing for it. The study recommends more diverse and comprehensive evaluation methods for AI systems.
Original article: Hacker News — Yapay Zeka Modelleri Pelikan Testinde Nasıl Performans Gösteriyor?