On Locality and Length-Generalization in Visual Reasoning

A striking feature of the human visual system is that itingests image information through a sequence of local glimpses, ratherthan a single global computation. This makes the human vision distinctlydifferent from most popular computer vision models in use today, whichingest images globally and in a single shot. A natural question thereforeis whether local, sequential vision models may provide any fundamentalcomputational benefits, in addition to being biologically more plausiblethan global models. In this work, we propose an answer to this questionfrom the perspective of state tracking and length generalization. Inspiredby recent studies of length generalization in language models, we studythe behavior of vision models trained on simple vision tasks that requirethe aggregation of local information across an image. Our experimentsreveal that, similar to language models, vision models can learn to ex-ploit global shortcuts and thereby fail to generalize over task length orcomplexity. We also show that recurrent vision policies based on strictlylocal perception can mitigate these failures, thereby allowing models togeneralize on these tasks. Our results show that local attention may be anessential overlooked requirement for robust compositional generalizationin vision tasks.