A scalable distributed framework for parallel—adaptive spacetime discontinuous Galerkin solvers with application to multiscale earthquake simulation

Madhukar, Amit

Permalink

https://hdl.handle.net/2142/117804 Copy

Description

Title

A scalable distributed framework for parallel—adaptive spacetime discontinuous Galerkin solvers with application to multiscale earthquake simulation

Author(s)

Madhukar, Amit

Issue Date

2022-12-01

Director of Research (if dissertation) or Advisor (if thesis)

Haber, Robert

Doctoral Committee Chair(s)

Haber, Robert

Committee Member(s)

• Fischer, Paul
• Elbanna, Ahmed
• Chandra Admal, Nikhil

Department of Study

Mechanical Sci & Engineering

Discipline

Mechanical Engineering

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• High Performance Computing
• Parallel Algorithms
• Geomechanics
• Spacetime discontinuous Galerkin
• Numerical solutions
• Earthquake rupture dynamics

Abstract

The causal Spacetime Discontinuous Galerkin (cSDG) method is a powerful numerical scheme for solving hyperbolic systems. cSDG solvers construct asynchronous, unstructured spacetime meshes that reflect the target system’s characteristic structure. When combined with spacetime discontinuous Galerkin bases, causal spacetime meshes localize the solution process to small clusters of spacetime elements called patches such that a localized implicit solution on each patch depends only on adjacent previously-solved predecessor elements and prescribed initial-boundary data. Starting with a simplicial space mesh with all vertices assigned initial-time coordinates, cSDG solvers advance a space-like front mesh through the spacetime analysis domain. Subsequent front-mesh configurations --- in the form of bumpy terrains with independent time coordinates for each vertex --- separate the previously-solved and unsolved parts of the analysis domain. Discrete increments of a vertex time coordinates advance the front locally subject to a causality constraint that requires every front configuration to be space-like. A simplicial spacetime mesh covering the local region between two front configurations defines a new patch on which a local, implicit and unconditionally stable cSDG problem is immediately solved. The new configuration then replaces its predecessor as the current front. The simulation terminates when the entire front-mesh passes the final time of the simulation. The localized cSDG solution procedure delivers element-wise balance properties and linear computational complexity in the number of patches. The cSDG method supports an extremely dynamic form of adaptive spacetime meshing capable of capturing fast-moving solution features such as sharp wavefronts and dynamic fracture. Adaptive meshing operations execute at the same patch-level granularity as the localized solution scheme. If a newly-solved patch fails its adaptive error tolerances, we discard the patch solution, refine the current front mesh below the failed patch, and restart the spacetime meshing process to generate simultaneous local refinement in both space and time. Other adaptive operations, including coarsening, edge flips, and vertex motion (for mesh smoothing, interface tracking, etc) are implemented by special patch configurations that perform the desired operation continuously in spacetime with no need for accuracy-limiting solution projections. The elimination of global time-step constraints, as in conventional time marching schemes, makes cSDG adaptive meshing particularly powerful. The cSDG method possesses a rich parallel structure in which the local operations --- spacetime patch construction, patch solution, and adaptive meshing --- form an embarrassingly parallel unit of computation. Only procedures required to prevent interference between patches involved in simultaneous parallel processing break the embarrassingly parallel structure. We show that standard parallel frameworks for PDE solvers are either incompatible with or would seriously degrade the strengths of cSDG solvers. In particular, the Bulk Synchronous Parallel (BSP) model (as in standard time marching) is incompatible with the cSDG method's asynchronous, fine-grained structure, and the Domain Decomposition Method (DDM) for parallelization requires expensive rebalancing computations that cannot keep pace with fine-grained cSDG adaptive meshing. We abandon the BSP model and the DDM and describe a task-based asynchronous framework for distributed parallel--adaptive cSDG solvers that is deployable on distributed-memory clusters and that matches the cSDG method's structure. It features patch-based parallelization, a latency tolerant scheme for accessing global front-mesh data (the only distributed data structure), and asynchronous probabilistic load and data balancing methods. We demonstrate the effectiveness of the parallel--adaptive cSDG method in multi-scale earthquake simulations, an application in which fault systems extending for hundreds of kilometers may require millimeter resolution to capture fine-scale response within rupture process zones. State-of-the art seismic simulation codes are unable to bridge the resulting extreme ranges of spatio-temporal scales, even on today’s largest supercomputers. We combine a dynamic contact model and a slip-weakening friction model with a cSDG model for linear elastodynamics to build a dynamic rupture model for fault dynamics. We verify our model with dynamic-rupture benchmark problems from the Southern California Earthquake Center (SCEC) and demonstrate the power of high-order cSDG models and adaptive spacetime meshing in bridging the broad range of spatio-temporal scales in these problems. While our results are in broad agreement with previous solutions, we uncover details of high-frequency response that are missing in previous solutions. We also discover a new dipole-like feature at fault branch points that can lead to separation under certain circumstances.

Graduation Semester

2022-12

Type of Resource

Thesis

Copyright and License Information

Copyright 2022 Amit Madhukar

Owning Collections