Offsite-Tuning: Transfer Learning without Full Model

Abstract

Transfer learning is important for foundation models to adapt to downstream tasks. However, many foundation models are proprietary, so users must share their data with model owners to fine-tune the models, which is costly and raises privacy concerns. Moreover, fine-tuning large foundation models is computation-intensive and impractical for most downstream users. In this paper, we propose Offsite-Tuning, an efficient transfer learning framework that can adapt billion-parameter foundation models to downstream data without data and full model exchanges. In Offsite-Tuning, the model owner sends a lightweight adapter and a lossy compressed emulator to the data owner, who then fine-tunes the adapter on the downstream data with the emulator's assistance. The fine-tuned adapter is then returned to the model owner, who plugs it into the full model to create an adapted foundation model. Offsite-Tuning is computationally more efficient and preserves two parties' privacy compared with existing fine-tuning methods that require access to the full model weights. We demonstrate the effectiveness of Offsite-Tuning on various large language and vision foundation models. Offsite-Tuning can achieve comparable accuracy as full model fine-tuning while being efficient and privacy-preserving, achieving 6.5x speedup and 5.6x memory reduction.

Citation

Team Members