Applying a Time-Varying Markov Decision Process to Deep Reinforcement Learning–Based Resource Allocation in Cloud Computing

Abstract

Cloud computing has become increasingly important for computing-intensive tasks such as geospatial analysis, heightening the importance of resource optimization. Classical scheduling algorithms including First Come First Serve (FCFS), Shortest Job First (SJF), and Round Robin (RR) often struggle with the dynamic nature of cloud workloads, creating longer-than-necessary job wait times and suboptimal resource utilization. More recently, studies have focused on deep reinforcement learning (DRL) methods for resource allocation, leveraging DRL’s ability to learn effective policies for complex, changing environments. However, most current research focuses on stationary Markov decision processes (MDPs), which assume that transition probabilities remain constant over time. While these processes provide a valuable approximation, real-world environments often feature nonstationary dynamics due to time-varying workloads and resource demand. Thus, a resource allocation policy that performs well during periods of low demand may perform poorly during peak usage. A DRL scheduler using a time-varying MDP was developed and trained on real-world monitoring logs from the MIT Supercloud system, with the intent of optimizing CPU and GPU resource utilization. This work contributes to ongoing efforts to develop autonomous system log analysis methods for Computing Infrastructure Digital Twins (CIDTs) (https://doi.org/10.3390/computers15070430). Initial results demonstrate an overall mean CPU utilization of 71.34% and a mean CPU utilization of 90.95% during peak-demand hours, suggesting that applying a time-varying MDP is a promising approach for enhanced resource efficiency in cloud computing systems.