SynthDocBench Exposes VLM Weaknesses in Long-Context Document Understanding

A new synthetic benchmark, SynthDocBench, systematically controls document length, layout, and question difficulty to isolate VLM failure modes. Testing seven frontier VLMs revealed sharp performance degradation with document length, systematic positional sensitivity, and breakdown of chart comprehension in long documents. The benchmark provides granular insights into model limitations that existing benchmarks like DocVQA and ChartQA cannot capture.