2024-08-11
Mini-ensemble low-rank adapters for parameter-efficient fine-tuning
Publication
Publication
Parameter-efficient fine-tuning (PEFT) is a popular method for tailoring pre-trained large language models (LLMs), especially as the models’ scale and the diversity of tasks increase. Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional, i.e., significant model changes can be represented with relatively few parameters. However, decreasing the rank encounters challenges with generalization errors for specific tasks when compared to full-parameter fine-tuning. We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential.The core idea is to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters. This can capture a significant degree of diversity among mini LoRAs, thus promoting better generalization ability. We conduct a theoretical analysis and empirical studies on various NLP tasks. Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks, which demonstrates the effectiveness of MELoRA.
Additional Metadata | |
---|---|
doi.org/10.18653/v1/2024.acl-long.168 | |
62nd Annual Meeting of the Association for Computational Linguistics (ACL) | |
Organisation | Distributed and Interactive Systems |
Ren, P., Chengshun, S., Shiguang, W., Mengqi, Z., Ren, Z., de Rijke, M., … Pei, J. (2024). Mini-ensemble low-rank adapters for parameter-efficient fine-tuning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (pp. 3052–3064). doi:10.18653/v1/2024.acl-long.168 |