Cloud-bursting and autoscaling for Python-native scientific workflows

Liu, Tingkai

Permalink

https://hdl.handle.net/2142/120266 Copy

Description

Title

Cloud-bursting and autoscaling for Python-native scientific workflows

Author(s)

Liu, Tingkai

Issue Date

2023-04-20

Director of Research (if dissertation) or Advisor (if thesis)

Kindratenko, Volodymyr

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• HPC
• Cloud
• Cloud-bursting

Abstract

In this work, the Ray framework is extended to enable automatic scaling of workloads on high-performance computing (HPC) clusters managed by SLURM and bursting to Cloud managed by Kubernetes. Compared to existing HPC-Cloud convergence solutions, this framework demonstrates advantages in several aspects: users can provide their own Cloud resource, framework provides the Python-level abstraction that does not require users to interact with job submission systems, and it allows a single Python-based parallel workload to be run concurrently across an HPC cluster and a Cloud. Applications in Electronic Design Automation and distributed Machine Learning are used to demonstrate the functionality of this solution in scaling the workload on an on-premises HPC system and automatically bursting to a public Cloud when running out of allocated HPC resources. The thesis focuses on describing the initial implementation and demonstrating novel functionality of the proposed framework, as well as identifying practical considerations and limitations for using Cloud bursting mode. The code of this framework has been open-sourced.

Graduation Semester

2023-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2023 Tingkai Liu

Owning Collections