Low-Rank Ternary Adaptation for Fine-Tuning Transformers

Ternary transformers offer extreme memory and compute ef-ficiency, but existing low-bit LoRA-based methods cannot directly fine-tune ternary weights. Current approaches either require dequantization,restoring low-bit base weights to higher precision to merge with adap-tation weight, or update only quantization parameters, preventing amerged model that remains ternary. We propose ternary multiplica-tive adaptation, which represents discrete updates of ternary weightssuch as sign flips or zeroing through a low-rank Kronecker factoriza-tion into two small ternary matrices applied element-wise to ternaryweights. This design is parameter-efficient and expressive, preserves theternary domain, and supports direct merging without dequantization.Experiments on six models across language and vision, including ternar-ized LLaMA-3 1B and 3B and a ternary ViT-B/16, demonstrate thatour method recovers much of the performance lost to quantization andoutperforms strong low-bit and ternary baselines. Code is available athttps://github.com/alexmanoo/ternary_adaptation.