[00:00] Intro: Research on multi-task learning in LLMs
[00:38] Balancing safety and performance in multilingual settings
[01:17] Model merging techniques explored
[02:16] Model merging outperforms data mixing
[02:49] Merging monolingual models improves multilingual capabilities
[03:28] Key ablation studies
[04:14] Safety and performance evaluation metrics
[04:52] Effectiveness variations across languages
[05:26] Safety model weight impact in linear merging
[06:00] Insights on merging and preference training
[06:25] Comparison to existing research
[06:56] Implications for LLM development
[07:29] Limitations and future research

Authors: Aakanksha, Arash Ahmadian, Seraphina Goldfarb-Tarrant, Beyza Ermis, Marzieh Fadaee, Sara Hooker
Affiliations: Cohere For AI, Cohere
Abstract: Large Language Models (LLMs) have been adopted and deployed worldwide for a broad variety of applications. However, ensuring their safe use remains a significant challenge. Preference training and safety measures often overfit to harms prevalent in Western-centric datasets, and safety protocols frequently fail to extend to multilingual settings. In this work, we explore model merging in a diverse multi-task setting, combining safety and general-purpose tasks within a multilingual context. Each language introduces unique and varied learning challenges across tasks. We find that objective-based merging is more effective than mixing data, with improvements of up to 8% and 10% in general performance and safety respectively. We also find that language-based merging is highly effective -- by merging monolingually fine-tuned models, we achieve a 4% increase in general performance and 7% reduction in harm across all languages on top of the data mixtures method using the same available data. Overall, our comprehensive study of merging approaches provides a useful framework for building strong and safe multilingual models.
Link: https://arxiv.org/abs/2410.10801

Ver todos los episodios