Unveiling Transferability in Trajectory Prediction via Latent Scene Embeddings

The growing availability of trajectory datasets has fueledmajor advances in data-driven motion prediction. Yet, models trained onone dataset often fail to generalize beyond their training domain as a resultof differences in scene layouts, agent behaviors, and sensing conditions. Aframework that learns latent representations of datasets and quantifiestheir similarity using distributional metrics is presented. This large-scalestudy covers 24 major datasets, including the most widely used motion-prediction benchmarks, and shows that the resulting transferability scoresstrongly correlate with cross-dataset model performance. The resultsprovide practical guidance for dataset selection, pretraining, and large-scale foundation models for motion prediction, paving the way towardmore generalizable and robust predictive systems.