Caption Bottleneck Models

Concept Bottleneck Models (CBMs) provide interpretabilityby routing predictions through a layer of human-understandable con-cepts. However, defining an optimal concept set for a specific datasetremains an open challenge. Existing approaches rely on expensive ex-pert annotations or LLM-generated lists based solely on class names.Even “open-vocabulary” variants typically depend on static concept sets,which restrict discovery and introduce label bias. Furthermore, tradi-tional CBMs often suffer from information leakage, where unmodeledvisual features bypass the bottleneck and compromise the integrity ofthe explanations. To overcome these limitations, we propose CaptionBottleneck Models (CaBM), a framework that circumvents the need forpredefined concept sets by replacing rigid concept layers with free-formnatural language. By representing images via LMM-generated captionsand training a classifier strictly on this text, CaBM ensures a leakage-freearchitecture by construction. Additionally, by analyzing the text classi-fier post-training, CaBM autonomously discovers high-quality, dataset-specific concepts. Our results across fine- and coarse-grained benchmarksdemonstrate that CaBM achieves competitive accuracy while preservinginterpretability without the constraints of external dictionaries or manuallabeling. Our code is available at https://github.com/bariscagliyan/CaptionBottleneckModels.