Vulnerability to Stability: Scalable Large Language Model in Queue-Based Web Service

Document Type

Article

Publication Date

2025

Publisher

Institute of Electrical and Electronics Engineers (IEEE)

Source Publication

IEEE 49th Annual Computers, Software, and Applications Conference (COMPSAC)

Source ISSN

2836-3795

Original Item ID

DOI: 10.1109/COMPSAC65507.2025.00129

Abstract

Large Language Models (LLMs) have demonstrated exceptional capabilities in the field of Artificial Intelligence (AI) and are now widely used in various applications globally. However, one of their major challenges is handling high-concurrency workloads, especially under extreme conditions. When too many requests are sent simultaneously, LLMs often become unresponsive which leads to performance degradation and reduced reliability in real-world applications. To address this issue, this paper proposes a queue-based system that separates request handling from direct execution. By implementing a distributed queue, requests are processed in a structured and controlled manner, preventing system overload and ensuring stable performance. This approach also allows for dynamic scalability, meaning additional resources can be allocated as needed to maintain efficiency. Our experimental results show that this method significantly improves resilience under heavy workloads which prevents resource exhaustion and enables linear scalability. The findings highlight the effectiveness of a queue-based web service in ensuring LLMs remain responsive even under extreme workloads.

Comments

Published as part of the proceedings of the IEEE 49th Annual Computers, Software, and Applications Conference (COMPSAC) 2025: 995-1000. DOI.

Link to Full Text

Share

COinS