Task-Agnostic Incremental Vision-Language Object Detection via Prompt Augmentation and Distribution-Aware Fusion

Incremental Vision-Language Object Detection (IVLOD) em-powers pre-trained open-vocabulary detectors to continuously learn newvisual concepts from sequential tasks without forgetting foundationalknowledge. However, existing methods predominantly rely on oracle taskidentities during inference, severely limiting their open-world practical-ity. In this paper, we formalize Task-Agnostic IVLOD (TA-IVLOD), amore realistic setting where such priors are unavailable, requiring themodel to simultaneously recognize classes from all learned tasks. Thisunconstrained setting exposes severe performance degradation due tocross-task semantic interference and parameter conflicts. To tackle thesechallenges, we propose TADA, a novel modular framework. Specifically,TADA introduces Stochastic Prompt Augmentation to mitigate semanticinterference via training-time noise injection, and Test-Time Distribution-Aware Fusion to dynamically weight class-specific LoRA experts, effec-tively resolving parameter conflicts. Extensive evaluations demonstratethat our method significantly outperforms baselines in standard IVLODand yields substantial improvements in the rigorous TA-IVLOD set-ting on the ODinW-13 benchmark, while effectively preserving zero-shot generalization on the MS COCO dataset. The code is available athttps://github.com/yonghanjiang/TADA.