CAR-MIL: Counterfactual Attention Regularization for Multiple Instance Learning

Multiple Instance Learning (MIL) is widely used for weaklysupervised learning, particularly in digital pathology, where fine-grainedannotations are costly. Most MIL methods aggregate instance featuresvia attention mechanisms. However, attention weights do not alwaysfaithfully reflect instance importance and may focus on spuriously cor-related regions. In this work, we propose CAR-MIL, a framework thatexplicitly guides attention learning through a counterfactual attentionregularization objective inspired by counterfactual explanations. Built ona standard attention-based MIL architecture, our approach introduces alightweight counterfactual attention branch trained to produce an alter-native prediction while remaining close to the factual attention distribu-tion. This encourages prediction changes to arise from minimal, struc-tured redistributions of attention, leading to more informative evidenceallocation. The resulting factual and counterfactual attention maps cap-ture complementary evidence: the former highlights regions supportingthe prediction, while the latter reveals regions whose reweighting wouldchallenge it. We evaluate our method on synthetic MIL benchmarks withinstance-level ground truth enabling controlled analysis of attention be-havior and on five digital pathology datasets across four tasks. CAR-MILmaintains competitive classification performance, with the largest gainsobserved on more challenging tasks, while improving attention reliability,demonstrating the benefits of integrating counterfactual explainabilityreasoning into attention learning.