AI · 1h ago
Study: Vision-Language AI Still Fails on Complex Social Scenes
A new study finds that vision-language models achieve near-human accuracy on simple benchmarks like MS-COCO but fail on complex social scenes. Researchers tested nine models from 2017-2025 on a new dataset of movie frames requiring multi-person interaction reasoning. While modern multimodal LLMs eliminated four of five error types, spatial dependence errors persist.
Meridian48 take
The study punctures inflated claims of human-level performance, showing that progress is partly an artifact of easy benchmarks and that real-world social understanding remains elusive.
Read the full reporting
Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models →
DEV Community
vision-language-modelsai-benchmarks