Learning with Bilevel-Minimax Optimization for Efficient and Reliable Transfer Attacks

Transfer-based adversarial attacks craft adversarial examplesusing surrogate models to mislead black-box victim models. Beyond per-turbation generation, transferability is fundamentally governed by thecoupling of initialization, surrogate adaptation, and gradient dynam-ics. We revisit this challenge from a Bilevel-Minimax perspective andinstantiate it in BMAT (Bilevel-Minimax Adversarial Transfer). Thebilevel formulation captures the dependency between initialization andperturbation, while the inner minimax problem promotes surrogate ro-bustness for cross-architecture generalization. Algorithmically, we designan integrated bottom-up solver that combines a Soft Weight Modulatorand an Implicit Gradient Approximator for ternary coupling interaction.We further provide theoretical insights into the optimization dynamicsof the proposed bilevel-minimax framework. Extensive experiments onclassification and segmentation benchmarks show that BMAT surpasses10+ strong baselines across 30+ victim models, improving both intra-and cross-architecture transfer, and yielding up to 2× mIoU reduction.Code is available at https://github.com/callous-youth/BMAT.