LLaMA-Omni, designed to improve the seamless interaction between speech and large language models (LLMs). This model integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder, allowing it to generate text and speech responses directly from speech instructions with minimal latency. To enhance the model's performance, the authors create a speech instruction dataset called InstructS2S-200K containing 200,000 speech instructions and corresponding speech responses. Experimental results demonstrate that LLaMA-Omni provides superior responses in both content and style compared to previous speech-language models, achieving a response latency of 226 milliseconds. Furthermore, the model's training process is efficient, requiring less than 3 days on 4 GPUs.

Ver todos los episodios