Optimizing I/O performance for high performance computing applications: from auto-tuning to a feedback-driven approach

Luu, Huong Vu Thanh

Permalink

https://hdl.handle.net/2142/78421 Copy

Description

Title

Optimizing I/O performance for high performance computing applications: from auto-tuning to a feedback-driven approach

Author(s)

Luu, Huong Vu Thanh

Issue Date

2015-04-22

Director of Research (if dissertation) or Advisor (if thesis)

• Winslett, Marianne
• Gropp, William

Doctoral Committee Chair(s)

Winslett, Marianne

Committee Member(s)

• Snir, Marc
• Ross, Robert

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• High Performance Computing (HPC)
• Input/Output (I/O)
• Parallel I/O
• Performance Analysis

Abstract

The 2014 TOP500 supercomputer list includes over 40 deployed petascale systems, and the high performance computing (HPC) community is working toward developing the first exaflop system by 2023. Scientific applications on such large-scale computers often read and write a lot of data. With such rapid growth in computing power and data intensity, I/O continues to be a challenging factor in determining the overall performance of HPC applications. We address the problem of optimizing I/O performance for HPC applications by firstly examining the I/O behavior of thousands of supercomputing applications. We analyzed the high-level I/O logs of over a million jobs representing a combined total of six years of I/O behavior across three leading high-performance computing platforms. Our analysis provides a broad portrait of the state of HPC I/O usage. We proposed a simple and effective analysis and visualization procedure to help scientists who do not have I/O expertise to quickly locate the bottlenecks and inefficiencies in their I/O approach. We proposed several filtering criteria for system administrators to find application candidates that are consuming system I/O resources inefficiently. Overall, our analysis techniques can help both application users and platform administrators improve I/O performance and I/O system utilization. In the second part, we develop a framework that can hide the complexity of the I/O stack from scientists without penalizing performance. This framework will allow application developers to issue I/O calls without modification and rely on an intelligent runtime system to transparently determine and execute an I/O strategy that takes all the levels of the I/O stack into account. Lastly, we develop a multi-level tracing framework that provides a much more detailed feedback for application’s I/O runtime behavior. These details are needed for in-depth application’s performance analysis and tuning.

Graduation Semester

2015-5

Type of Resource

text

Permalink

http://hdl.handle.net/2142/78421

Copyright and License Information

Copyright 2015 Huong Vu Thanh Luu

Owning Collections