Artificial Intelligence (AI)-driven defect inspection is pivotal in industrial manufacturing. However, existing inspection systems are typically designed for specific industrial products and struggle with diverse product portfolios and evolving processes. Although some previous studies attempt to address object dynamics by storing embeddings in the reserved memory bank, these methods suffer from memory capacity limitations and object distribution conflicts. To tackle these issues, we propose the Incremental Unified Framework (IUF), which integrates incremental learning into a unified reconstruction-based detection method, thus eliminating the need for feature storage in the memory. Based on IUF, we introduce Object-Aware Self-Attention (OASA) to delineate distinct semantic boundaries. We also integrate Semantic Compression Loss (SCL) to optimize non-primary semantic space, enhancing network adaptability for new objects. Additionally, we prioritize retaining the features of established objects during weight updates. Demonstrating prowess in both image and pixel-level defect inspection, our approach achieves state-of-the-art performance, supporting dynamic and scalable industrial inspections. Our code will be released.