Petascale Research in Earthquake System Science on Blue Waters (Presson)

Petascale Research in Earthquake System Science on Blue Waters (PressOn)

(OCI-0832698)

Project Annual Report for Performance Period:

1 October 1 2012 – 30 September 2013

Principal Investigator:

Thomas H. Jordan – University of Southern California – Earth Sciences

Co-PI:

Jacobo Bielak – Carnegie Mellon University – Civil Engineering

1. What are the major goals of the project?

The PRAC review panel approved the PetaURB objectives defined in our PressOn PRAC proposal. Our PressOn proposal defines the PetaURB project objectives as:Develop methods, capability, and software for high fidelity and physics-based simulations of entire urban regions to assess the engineering impacts of large magnitude earthquakes on buildings, transportation systems, and underground civil infrastructure.

To meet these PetaURB objectives, SCEC must bridge the gaps between seismology and civil engineering, applying the latest computational seismological capabilities to study and assess the impact of strong ground motions on a built environment.

2. What was accomplished under these goals (you must provide information for at least one of the 4 categories below)?

2.1) Major activities;

SCEC has been actively developing physics-based seismological modeling codes in order to study the impact of large earthquakes on the built environment. We have continued to collaborate with building engineers to ensure that our PressOn research results meet the needs of the earthquake engineering community working to assess the engineering impact of large magnitude earthquakes. Many of our active developments and recent simulations are related to California earthquakes. However, the tools and techniques we have developed are applicable to other regions in the world.

The computational research areas for our SCEC PressOn PRAC project continues to include: (1) 3D velocity model development, (2) dynamic rupture and Earthquake Rupture Forecast development, (3) high frequency earthquake simulations, (4) ensemble probabilistic seismic hazard analysis (PSHA) earthquake simulations, (5) impact of ground motion on built environment. We worked in each of these areas during this performance period. Here is a summary of current project usage on Blue Waters:

Project Group (Jordan): Project ID: (jmz)

Allocated Node Hours: 3,394,000

Used Node Hours: 1,752,794

Date: 30 Oct 2013

Current SCEC PressOn Blue Waters Usage:

Users: / Software / Node Hours
Rupture and ERF:
Xu: / RSQSim / 667,688
Shi: / SORD / 68,408
Withers / AWP-ODC / 41,096
High-FreqGround Motion:
Patrices: / Hercules - GPU / 247
Yfcui: / AWP-ODC-GPU / 26,505
Poyraz: / AWP-ODC-GPU / 135,492
Gill: / UCVM / 1,146
Olsen / AWP-ODC / 79,063
PSHA:
Scottcal / CyberShake / 352,690
Building Response:
Taborda: / Hercules / 292,876
Karaoglu / Hercules / 87321
Total Used: / 1,752,532
Total Remaining: / 3,394,000 / 1,641,206

2.3) Significant results, including major findings, developments, or conclusions;

Development of Software and System Capabilities

The SCEC PressOn researchers developed dynamic rupture codes scalable (to thousands of cores), that can incorporate complex (non-Cartesian) fault geometry, and advanced thermomechanical models [Shi, Z., S.M. Day, and G. Ely (2012))].

We developed dynamic rupture simulations that can incorporate multiple physical models such as frictional breakdown, shear heating, porothermoelastic flow (and the resulting effective normal-stress fluctuations), as well as multiscale fault roughness [Roten, D., K. B. Olsen, J. C. Pechmann (2012)].

We used SORD to perform three-dimensional numerical calculations of dynamic rupture along non-planar faults to study the effects of fault roughness (self-similar over three orders of magnitude in scale length) on rupture propagation and resultant ground motion [Shi, Z., and S. M. Day (2013)].

We continued Hercules development incorporating free-surface topography and the influence of the built environment in the modeling and simulation scheme in Hercules [Restrepo, D. (2013); Isbiliroglu, Y., R. Taborda, and J. Bielak, J. (2013), Restrepo, D. (2013)]. This team developed a finite-element based methodology that uses special integration techniques to account for an arbitrary free-surface boundary in the simulation domain but preserves the octree- based structure of the code, and thus does not have a significant effect on performance.

SCEC PressOn researchers have improved code scientific and computing performance including AWP [Christen, M., O. Schenk, and Y. Cui (2012)], Hercules [Taborda, R. and Bielak, J. (2013a)], and SORD [Ely, G., (2013a)].

We also developed GPU versions of AWP-ODC [Cui, Y., E. Poyraz, K. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin, C. Guest, D. Choi, A. Chourasia, Z. Shi, S. Day, P. Maechling, and T. H. Jordan (2013)], Hercules, and CyberShakeSGT code [Cui, Y., E. Poyraz, J. Zhou, S. Callaghan, P. Maechling, T. H. Jordan, L. Shih, and P. Chen (2013)].

Improved Simulation Methodsby Adding new Physics into Simulation Codes:

SCEC PressOn researchers used Blue Waters to develop rough fault model simulations [Withers, K., K. B. Olsen, S. Shi, S. M. Day, and R. Takedatsu (2013)]. We used the SORD code [Ely, G., (2013c)] as a tool for dynamic simulation of geometrically and physically complex ruptures. To do so, we integrated high-speed frictional weakening (in a rate- and state-dependent formulation) into the code. This integration was done using a method that time-staggers the state and velocity variables at the split nodes, producing a stable, accurate and very efficient solution scheme. We also addedthe Drucker-Prager formulation of pressure-dependent plastic yielding into SORD, with added viscoplastic terms to suppress strain localization. The resulting code was successfully tested using SCEC rupture dynamics benchmarks. We also implemented and successfully tested a scheme for the generation of SORD meshes for power-law rough faults.

SORD dynamic rupture simulations can now model seismic wave excitation up to ~10 Hz with rupture lengths of ~100 km, permitting comparisons with empirical studies of ground- motion intensity measures of engineering interest.

The SCEC-developed, Cuda-language, wave propagation code called AWP-ODC-GPU [Cui, Y., K.B. Olsen, J. Zhou, P. Small, A. Chourasia, S.M. Day, P.J. Maechling, T.H. Jordan. (2012)] achieved sustained Petaflops [Cui, Y., E. Poyraz, K. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin, C. Guest, D. Choi, A. Chourasia, Z. Shi, S. Day, P. Maechling, and T. H. Jordan (2013)] and was used to run a 10Hz deterministic ground motion simulation [Withers, K., K. B. Olsen, S. Shi, S. M. Day, and R. Takedatsu (2013)] using a high frequency earthquake rupture produced by a dynamic rupture on a rough fault [Shi, Z., S.M. Day, and G. Ely (2012)], in a velocity model containing small scale heterogeneities [Olsen, K. B., W. Savran, B. H. Jacobsen (2013)].

2.4) Key outcomes or other achievements.

SCEC PressOn researchers used the earthquake rupture simulator RSQSim to develop a new theoretical approach for analyzing fault rupture synchronicity [Milner, K.R., Thomas H. Jordan (2013)].The main object of this analysis is the complete set of interevent time differences, which can be characterized in terms of the auto-catalog density function (ACDF) and the cross-catalog density function (CCDF).

SCEC PressOn researchers used Hercules software on Blue Waters to generate synthetics up to 5Hz+ and quantify the spatially varying fit between observation and simulations using engineering-oriented goodness-of-fit algorithms [Taborda, R. and Bielak, J. (2013a)]. We evaluated the amplification effects of the near-surface material (Vs<500m/s). The CMU group performed a set of simulations for the Mw 5.4 2008 Chino Hills, California earthquake using the various Southern California Velocity Models (CVM-S, CVM-H and CVM-H+GTL)[Taborda, R. and Bielak, J. (2013b,c, 2014)]. The simulations were designed to produce a valid representation of the ground motion up to a maximum frequency of 4 Hz [Taborda, R. and Bielak, J. (2013d)]. We compared the results of simulations of the Chino Hills earthquake with seismic records obtained from Southern California strong motion networks. In total, we compared simulation results against data in 336 stations. The quality of the match between the actual records and the simulated synthetics was measured in terms of a commonly used engineering-oriented goodness-of-fit (GOF) criterion.

SCEC PressOn researchers calculated four new CyberShake hazard models [Callaghan, S., Maechling, P., Juve, G., Vahi, K., Graves, R. W., Olsen, K. B., Gill, D., Milner, K., Yu, J. and Jordan, T. H. (2013)] using NSF and XSEDE resources Blue Waters and Stampede. Theseseismic hazard models were computed using different HPC codes (AWP-ODC, and RWG), different velocity models (CVM-S4 and CVM-H11.9), and the most recent Graves & Pitarka (2010) rupture generator (GP-10).

Using these CyberShake 13.4 results, researchers generalized the formulation of PSHA to accommodate simulation-based hazard models [Wang, F., Jordan, T H. (2012)], and from this generalization they developed a ground motion analysis method called Averaging-Based Factorization, which allows CyberShake hazard models to be decomposed into components that can be quantitatively compared with each other and with empirical hazard models, such as the Next Generation Attenuation (NGA) ground motion prediction equations [Wang, F., Jordan, T H. (2013)]. These comparisons have been used to examine the dependences of basin effects, directivity effects, and directivity-basin coupling on the structure of the pseudo-dynamic rupture models and the Community Velocity Models (CVMs) used in the large-scale simulations, including theCyberShake Study 13.4 run in April 2013 [Wang, F. et al., 2013].

Caltech seismologist MarenBoese used PressOn CyberShake results to train a neural network to improve rapid ground motion estimates for use in Earthquake Early Warning (EEW) application [Boese, M., Graves, R W., Callaghan, S., Maechling, P J. (2012)], an example of how PressOn HPC results were used in an unanticipated, non-HPC, broad impact application.

3. What opportunities for training and professional development has the project provided?

By providing access to Blue Waters, and travel funds to participate in Blue Waters meetings, our PRAC proposal has provided opportunities for training and professional development to project members.

Two post-doctoral researchers as well as three geoscience students were employed on the project within the SDSU and SDSC project teams. The post-doctoral researchers and students obtained advanced training in high- performance computing, rupture dynamics and 3D ground motion simulations during the project. They furthermore presented their work at international conferences and gained important technical communication skills during the project. UCSD PhD students Jun Zhou and EfecanPoyraz were trained through this project and the project mentored Jun Zhou during development of his Ph.D thesis project.

At Carnegie Mellon University, postdoctoral mentoring was part of the activities to help prepare Dr. Ricardo Taborda for an academic career. As a postdoctoral fellow in the Computational Seismology Laboratory at Carnegie Mellon (CMU), Dr. Taborda assisted Co-PI Bielak on advising three graduate students, writing journal publications and preparing poster and oral presentations (see publications). At CMU, Taborda has been particularly involved in co- advising the Ph.D. thesis work of graduate student YigitIsbiliroglu, whose topic of research is a continuation of Taborda’s Ph.D. work on the effects of the built environment on the ground motion during strong earthquakes and the coupled soil-structure interaction effects on the dynamic response of building clusters. Taborda has also served as the primary liaison between SCEC/IT group and the Quake Group at CMU, and has been closely involved with the development of the Unified California Velocity Model (UCVM). He has used datasets generated using UCVM to conduct his own research on the validation of the various seismic velocity models available for Southern California (CVM-S, CVM-H). During the spring of 2013, Taborda interviewed at several universities and secured a position as a new Assistant Professor at the University of Memphis (U of M), starting August 2013. At the U of M, Taborda has joined the faculty of the Civil Engineering Department in the School of Engineering and has a joint tenure-track appointment with the Center for Earthquake Research and Information (CERI). CERI is a University of Memphis Center of Excellence. We expect that Taborda will continue to collaborate with SCEC and CMU from CERI. Two other graduate students, HaydarKaraoglu and DoriamRestrepo participated in the research activities at CMU. DoriamRestrepo has successfully defended his Ph.D. thesis and will be joining the faculty at the University of EAFIT in Medellin, Colombia, and HaydarKaraoglu is expected to complete his Ph.D. studies in May 2014.

At USC, one of Project PI Tom Jordan’s graduate students participated in the project: F. Wang. Wang graduated with a PhD in September, and he is now employed in the position of Scientist at AIR Worldwide, applying his SCEC-based research to hazard and risk analysis in the commercial sector.

4. How have the results been disseminated to communities of interest

SCEC PressOn project members have presented and discussed our work in a series of geoscientific and computational science meeting and workshops during the project including the following:

1.SCEC Annual Meeting, Sept 9-12, 2013 Palm Springs, CA

2.IASPEI Joint Assembly, July 25, 2013 Gothenburg, Sweden

3.SCEC CME Project Meeting, 2 June 2013, Palm Spring CA

4.SSA April 17-19, 2013 Salt Lake City, Utah

5.Ground Motion Simulation Validation Technical Activity Group Workshop, April 3, 2013

6.Organizational Meetings of a SCEC Committee for Utilization of Ground Motion Simulations, April 3, 2013 Los Angeles CA

7.CIG-QUEST-IRIS Workshop: Seismic Imaging of Structure and Source. July 14-17, 2013. University of Alaska Fairbanks

8.Fall AGU, Dec 3-7, 2012 – San Francisco, CA

9.Global Earthquake Model Annual Meeting, Dec 9-13, 2012– Pavia, Italy

10.Lawrence Livermore National Laboratory HPC in Geophysics Meeting, Nov 8-9, 2012, LLNL, CA

11.Invited Speaker, HPC China Workshop at SC’12, Nov 13, 2012, International Workshop on CO-DESIGN, Beijing, Oct 23-25, 2012

12.SCEC Ground Motion Simulation Validation Workshop Sept 9, 2012, Palm Springs, CA

13.REAKT Meeting Oct 8-10, 2012 Potsdam Germany

We have also worked with NSF and other communications experts to develop scientific summaries of our recent accomplishments and impact. Several press articles, including NSF Discoveries articles, included descriptions of SCEC’s use of NSF computer during year 1. Links to these online press articles include the following:

1.

2.

3.

4.

5.

6.

7.

8.

14.

15.

16.

17.

SCEC has hosted a series of computational workshops on ground motion simulation and modeling issues including a Rupture Dynamic Code validation workshop (March 2013), a SCEC Ground Motion Simulation Validation (GMSV) Technical Activity Working Group (April 2013) meeting, and SCEC Committee for Utilization of Ground Motion Simulations (UGMS) (April 2013) meeting. Details from these meetings are posted on the public SCECpedia wiki.

SCEC researchers make use of wiki’s to communicate our research among ourselves. By default, research communications are open. Many SCEC PressOn research efforts, including the wave propagation simulation work are described in some detail on the SCEC wiki with the following home page: Google analytics for this site from 1 October 2012 through 30 September 2013 show 7,797 visits, from 3,006 unique visitors, with 29,896 pageviews, averaging 3.83 pages/visit.

5. What do you plan to do during the next reporting period to accomplish the goals?

By our current estimates, we have completed approximately 50% of our original SCEC PressOn PetaURB PRAC research plan. We are pushing deterministic seismic hazard simulations to 10Hz, while developing the physics-based models, and the computational tools to perform these demanding simulations. We are also working to improve our understanding ofbuilding response to ground motions so the engineering community can utilize our improved seismic hazard models as quickly as possible.

The table below summarizes our remaining SCEC PRAC simulation plan that we believe is possible without a significant increase in our current Blue Waters allocation. This table shows scientific work areas, the codes we expect to use, and the estimated Node Hours for each research calculation. The total remaining

Proposed SCEC PressOn Blue Waters Simulation Plan (Oct 2013 – Sept 2014):

Project / Software / Node Hours
Rupture and ERF:
Chino Hills SRF 10Hz / SORD / 50,000
Whittier Narrows SRF 10Hz / SORD / 50,000
High-FreqGround Motion:
10Hz Whittier Narrows / Hercules - GPU / 312,000
10Hz Whittier Narrows / AWP-ODC-GPU / 437,000
PSHA:
SoCal PSHA 0.5Hz 300 sites 2 components CVM-S4 / CyberShake – GPU / 250,000
SoCal PSHA 0.5Hz 300 Sites
3 components CVM-SI.26 / CyberShake – RWG / 450,000
DRM Building Response:
LA Region DRM Study / Hercules / 150,000
Required Node Hrs: / 1,699 ,000

6. Products - What has the project produced?

Publication List:

1. Boese, M., Graves, R W., Callaghan, S., Maechling, P J. (2012) Site-specific Ground-Motion Predictions for Earthquake Early Warning in the LA Basin using CyberShake Simulations Abstract S53B-2498 Poster presented at 2012 Fall Meeting, AGU, San Francisco, Calif., 3-7 Dec.
2. Callaghan, S., Maechling, P., Gideon Juve, Karan Vahi, Robert W. Graves, Kim B. Olsen, David Gill, Kevin Milner, John Yu and Thomas H. Jordan (2013) Running CyberShake Seismic Hazard Workflows on Distributed HPC Resources, SCEC Annual Meeting 2013, abstract 195, Sept 8 – 11, 2013, Palm Springs, CA
3. Callaghan, S., Maechling, P J., Milner, K., Graves, R W., Donovan, J., Wang, F., Jordan, T H (2012) CyberShake: Broadband Physics-Based Probabilistic Seismic Hazard Analysis in Southern California Abstract S51A-2405 presented at 2012 Fall Meeting, AGU, San Francisco, Calif., 3-7 Dec.
4. Chourasia, A., Zhou, J., Cui, Y., Choi, DJ, Olsen, K. (2012) Role of visualization in porting a seismic simulation from CPU to GPU architecture (Visualization Showcase), XSEDE’12, Chicago, July 16-20, 2012.
5. Christen, M., O. Schenk, and Y. Cui (2012) PATUS for Convenient High-Performance Stencils: Evaluation in Earthquake Simulations, Technical Paper, SC12, Salt Lake City, Nov 10-16, 2012.
6. Cui, Y., E. Poyraz, K. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin, C. Guest, D. Choi, A. Chourasia, Z. Shi, S. Day, P. Maechling, and T. H. Jordan (2013), Physics-based seismic hazard analysis on petascale heterogeneous supercomputers, SC13, Denver, Nov 17-22, 2013 (accepted for publication)
7. Cui, Y., E. Poyraz, J. Zhou, S. Callaghan, P. Maechling, T. H. Jordan, L. Shih, and P. Chen (2013), Accelerating CyberShake Calculations on the XK7 Platform of Blue Waters, Extreme Scaling Workshop, Denver, August 15-16, 2013
8. Cui, Y., K.B. Olsen, J. Zhou, P. Small, A. Chourasia, S.M. Day, P.J. Maechling, T.H. Jordan. (2012). Development and optimizations of a SCEC community anelastic wave propagation platform for multicore systems and GPU-based accelerators, Seism. Res. Lett. Seism. Res. Lett. 83:2, 396.
9. Donovan, J., T. H. Jordan, and J. N. Brune (2012). Testing CyberShake using precariously balanced rocks, 2012 Annual Meeting of the Southern California Earthquake Center, Palm Springs, Abstract 026, September, 2012.
10. Donovan, J., and T.