This July 2025 paper discusses advanced memory optimization techniques for Large Language Models (LLMs), particularly focusing on KV cache management in multi-tenant serving environments. The primary subject, MIRAGE, introduces parameter remapping, a novel method that dynamically repurposes GPU memory allocated for model parameters to expand KV cache capacity, outperforming traditional CPU-offloading and KV cache swapping by reducing latency and increasing throughput. Complementary research highlights challenges in on-device LLM deployment and proposes solutions like quantization (AWQ) for model compression and two-level scheduling (FineServe, Nexus) for efficient GPU sharing to mitigate memory fragmentation and improve performance. Overall, the papers underscore the critical need for innovative memory management to address the growing memory demands of LLMs and enhance their inference serving efficiency across diverse hardware configurations.Source:https://www.researchgate.net/publication/393724496_MIRAGE_KV_Cache_Optimization_through_Parameter_Remapping_for_Multi-tenant_LLM_Serving

Ver todos los episodios