BeyondSight: Object Permanence for End-to-End Autonomous Driving

Autonomous driving operates in partially observable envi-ronments where actors may become fully occluded by other vehicles orinfrastructure. Most end-to-end driving systems implicitly couple actorexistence to instantaneous observations, causing actor hypotheses to de-grade or disappear during prolonged occlusion and removing potentiallycritical agents from downstream prediction and planning. We introduceBeyondSight, a permanence-aware end-to-end driving framework thatdecouples actor existence from observability by maintaining persistentactor hypotheses over time. BeyondSight propagates actor queries tem-porally and updates them with observation-conditioned evidence, en-abling joint perception, prediction, and planning to reason about ac-tors even when they are temporarily unobservable. To enable principledtraining and evaluation of persistence-aware models, we further introducenuScenes-Permanence, an extension of nuScenes that provides super-vision and observability-conditioned evaluation for unobservable actors.Experiments show that BeyondSight substantially improves reasoningunder occlusion, increasing detection performance for unobservable ac-tors from 0 to 0.249 mAP while reducing planning error from 0.61 to0.54 L2avg . These results highlight object permanence as an importantmodeling principle for robust end-to-end autonomous driving.