Visuospatial neglect is a frequent and disabling consequence of injuries to the right hemisphere. Patients with neglect show signs of impaired attention for left-sided events, which depends on dysfunction of fronto-parietal networks. After unilateral injury, such as stroke, these networks and their contralateral homologs can reorganize following multiple potential trajectories, which can be either adaptive or maladaptive. This article presents possible factors influencing the profile of evolution of neglect towards recovery or chronicity, and highlights potential mechanisms that may constrain these processes in time and space. The integrity of white matter pathways within and between the hemisphere appears to pose crucial connectivity constraints for compensatory brain plasticity from remote brain regions. Specifically, the availability of a sufficient degree of inter-hemispheric connectivity might be critical to shift the role of the undamaged left hemisphere in spatial neglect, from exerting maladaptive effects, to promoting compensatory activity.