SCEC INCITE OLCF Project Progress Report, 2015

High Frequency Ground Motion Simulations for Seismic Hazard AnalysisLead PI: Jordan

SCEC INCITE OLCF Project Progress Report, 2015

Title: High Frequency Ground Motion Simulations for Seismic Hazard Analysis

PI and Co-PI(s): Thomas H. Jordan, Jacobo Bielak, Kim Olsen, Yifeng Cui, Po Chen, Ricardo Taborda, Philip Maechling

Applying Institution/Organization: University of Southern California

Project Duration: 2 Years (Jan, 2015 - Dec, 2016)

Number of Processor Hours awarded on Titan: 119M core-hours

Executive Summary:Economic exposure to earthquake devastation has increased significantly over the last few decades due to massive growth of urban areasin seismically active regions. Physics-based modeling and simulation provide pathways to a more accurate representation of earthquake systems, from the rupture at the fault to the response of the built environment. Our goal is to produce earthquake simulations with sufficient bandwidthto be useful across a wide range of engineering applications. Our specific INCITE projectobjectivesare twofold: (a)Use DOE supercomputers to produce a comprehensive, physics-based hazard model for the Los Angeles region valid up to seismic frequencies of 1Hz, and (b)Extend realistic earthquake simulations above the 1-Hz frequencybarrier by incorporating new aspects of earthquake physics. Our achievements mark substantial progress towards these goals:

a)A 1-Hz urban seismic hazard model for the Los Angeles region has been completed (Figure1). The new model, which comprises more than 300 million synthetic seismograms sampling the Uniform California Earthquake Rupture Forecast, was computed from a new high-resolution image of crustal structure derived using full-3D tomography (CVM-S4.26). It will be registered into the USGS Urban Seismic Hazard Mapping Project, and the results will be submitted for use in the 2020 update of the Recommended Seismic Provisions of the National Earthquake Hazards Reduction Program.

b)High-frequency simulations (out to 5Hz) have been performed on the OLCF Titan supercomputer using GPU-optimized finite-difference and finite-element codes that include frequency-dependent attenuation, small-scale near-surface heterogeneities, tomography, and a nonlinear dissipation in the near-fault and near-surface regions. These simulations set the stage for the ground motion prediction modeling at frequencies beyond 1Hz.

Accomplishments led by Dr. Yifeng Cui in thedevelopment of GPU-enabled wave-propagation codes were recognized with NVIDIA’s 2015 Global Impact Award.

Project Accomplishments: SCEC's research team used the OLCF Titanand NCSABlue Waterssupercomputersto perform CyberShake Study 15.4 (initiated in April, 2015). This computationdoubled the maximum seismic frequency represented in the Los Angeles urban seismic hazard model, from 0.5Hz to 1Hz.Seismic hazard curves were derived from large ensembles of seismograms at frequencies below this maximum for 336 surface sitesdistributed across the Los Angeles region. This new probabilistic model uses refined earthquake rupture descriptions through revisions to the conditional hypocenter distributions and the conditional slip distributions. The CS15.4 model provides new seismic hazard information of interest to broad impact customers of CyberShake, including seismologists, utility companies, and civil engineers responsible for California building codes. The new model, which samples the complete Uniform California Earthquake Rupture Forecast, will be registered into the USGS Urban Seismic Hazard Mapping Project (

The GPU-based anelastic wave propagation AWP-ODC software was used to run CPU-based post-processing calculations that synthesized over 300 million seismograms. In Study 15.4, SCEC utilized approximately 200 pilot jobs to run CyberShake tasks on Titan resources. Over 80% of the node-hours burned on Titan were from jobs which ran on 25% or more of the machine. Approximately 200 TB of SGT data was transferred from Titan to Blue Waters automatically as part of the workflow. On Titan, the accelerated calculations of the GPU Strain Green Tensor (SGT) implementation is 6.3 times more efficient than the CPU implementation, which saved us 2 million node-hours over the course of the study. Our GPU development was recognized with NVIDIA’s 2015 Global Impact Award. “The full three-dimensional treatment of seismic-wave propagation has the potential to improve seismic hazard analysis models considerably, and that is where the accelerating technology is particularly helpful at this moment,” said Thomas Jordan, director of SCEC. “With GPU computing power we’re gaining insight as to how the ground will move in high-risk areas, and how we can better plan for the aftermath of a major event.”

The SCEC finite-element wave propagation solver, Hercules, which integrates an efficient octree-based hexahedral mesh generator with an explicit FE formulation, has been optimized on Titan this year achieving near perfect strong and weak scaling. Its GPU capabilities are currently being used in verificationand validation studies for the 2014 Mw 5.1 La Habra earthquake on Titan, to test the accuracy of the code compared to other codes, and to examine how close the predicted ground motions are to observations.

We have implemented non-associated Drucker-Prager nonlinear rheology following the return map algorithm in the scalable AWP-ODC code, and we have used this code to model ground motions from the M7.8 ShakeOut scenario source description. This work accounts for the limited strength of crustal rocks; i.e., to simulate the absorption of rupture energy by permanent rock deformation. Our results suggest that this nonlinear behavior could reduce previous simulation-based predictions of expected ground motion velocity in the Los Angeles basin during a large-magnitude event on the southern San Andreas Fault by 30 to 70 percent. Nonlinear material response occurs in soft soils near the surface, typically reducing high-frequency (>1 Hz) shaking that controls damage to low- and mid-rise buildings. Our simulations show that nonlinear response in crustal rocks may also reduce the amplitudes of long-period surface waves that pose a hazard to high-rise buildings, implying less destruction than previously anticipated. Although more research will be needed to quantify the impact of these findings on damage and casualty estimates for future large-magnitude earthquakes on the San Andreas Fault, the study pioneers moreaccurate earthquake scenarios based on better representations of the nonlinearity in the Earth's crust.

We have implemented realistic attenuation structure (frequency-dependent Q, or Q(f)) in the GPU-based AWP-ODC code (Withers et al., 2015). Tests using the 2008 Mw 5.4 Chino Hills earthquake indicate that Q(f) generally fits the strong motion data better than for constant Q models for frequencies over 1-Hz, which becomes more and more important as the distance increases from the fault. We also found that media heterogeneity reduces the within-event variability to that for observations and is thus important to characterize the ground motion.

Realistic ground-motion simulations require highly accurate crustal structural models. A significant portion of the awarded computational resources was used to construct full-3D, high-resolution crustal seismic velocity models in the Central California region and also in the statewide California through full-3D seismic waveform tomography (F3DT) (Lee et al. 2014ab). F3DT represents the latest development in seismic tomography techniques. Its application to seismic data recorded in Southern California has yielded a new community velocity model for the region, CVM-S4.26, which has unprecedented resolution of crustal structure. CVM-s4.26 is the 3D structural model used in the CyberShake 15.4 study.

We have further improved the computational efficiency of our F3DT workflow on ALCF Mira and are now applying F3DT to Central California and statewide. As of June, 2015, we have carried out 5 F3DT iterations for Central California and 3 F3DT iterations for the statewide California. Our improved Central California velocity model provides substantially better fit to over 12,000 seismic waveforms at frequencies up to 0.2 Hz and shows interesting small-scale structures in the upper to mid crust that agree with local geology and other independent geophysical evidence. Our latest statewide velocity model significantly improves the fit to over 27,000 waveforms at frequencies up to 0.1 Hz, and ithas revealed new structural features in themid to lower crust that are consistent with our understanding of the geotectonic development in California. More F3DT iterations will be carried out for both Central California and statewide. Gradual improvements in our velocity models have allowed us to incorporate an increasing volume of observed seismograms into our F3DT workflow, which is allowing us to resolve finer structural details with higher accuracy.

Impact of Research:The San Andreas fault system is prone to major earthquakes, yet Los Angeles has not experienced a major quake since its urbanization in the early twentieth century. Data for the region are available from smaller quakes, but such information doesn’t give emergency officials and structural engineers the information they need to prepare for a quake of magnitude 7.5 or bigger. CyberShake Study 15.4 represents a major milestone in physics-based PSHA for Southern California. The performance of the code and improved workflow management, combined with the new physics it models (e.g., fault roughness, small-scale heterogeneities, frequency-dependent attenuation, near-surface nonlinearities), take physics-based seismic hazard analysis to a new level and pioneer the use of Petascale heterogeneous computing resources for ground motion simulations used in building engineering design and evaluation.

The reduction of peak velocities in our models caused by mostly shallow, near-fault nonlinear effects may have important implications for the scaling of ground motion intensities between surface-rupturing and buried earthquakes. Our nonlinear simulation results show that nonlinearity in the fault zone is important even for conservative values of cohesion, suggesting that current simulations based on a linear behavior of rocks are over-predicting the level of ground motion in the Los Angles sedimentary basins during future large earthquakes on the southern San Andreas Fault, and possibly for other large earthquake scenarios. This will have far-reaching implications on earthquake emergency planning scenarios that are based on ground motions predictions, such as the damage scenario of the 2008 Great California ShakeOut. The addition of statistical models of near-surface small-scale heterogeneities has enabled us to capture the “within-event” variability of earthquakes more accurately, providing models that can be used to improve physics-based seismic hazard analysis.

Publications:

Isbiliroglu, Y., R. Taborda and J. Bielak (2015) Coupled soil-structure interaction effects of building clusters during earthquakes. Earthquake Spectra. Vol. 31, No. 1, 463-500, Feb 2015.

Donovan, J. (2015),Forecasting Directivity in Large Earthquakes in Terms of the Conditional Hypocenter Distribution, PhD Thesis, University of Southern California, 154 pp.

Jordan, T. H. (2015), An effective medium theory for three-dimensional elastic heterogeneities, Geophys. J. Int., submitted Mar 29, 2015.

Lee E.-J., P. Chen, T. H. Jordan, P. B. Maechling, M. A.M. Denolle and G. C. Beroza (2014a), Full-3D tomography for crustal structure in Southern California based on the scattering-integral and the adjoint-wavefield methods, J. Geophys. Res., 119, 6421-6451, doi:10.1002/2014JB011346.

Lee, E.-J., P. Chen, and T. H. Jordan (2014b), Testing waveform predictions of 3D velocity models against two recent Los Angeles earthquakes, Seismol. Res. Lett.,85, 1275-1284, doi:10.1785/0220140093.

Lozos, J., K.B. Olsen, J. Brune, R. Takedatsu, R. Brune, and D.D. Oglesby (2015), Broadband ground motions from dynamic models of rupture on the northern San Jacinto fault, and comparison with precariously balanced rocks, Bull. Seismol. Soc. Am., 105. (in press), doi: 10.1785/0120140328.

Olsen, K.B. and R. Takedatsu (2015), The SDSU Broadband Ground-Motion Generation Module BBtoolbox Version 1.5, Seism. Res. Letter, 86, 1, 81-88.

Poyraz, E., H. Xu and Y. Cui (2014), Application-specific I/O Optimizations on Petascale Supercomputers, Proceedings of International Conference on Computational Science, Elsevier, 29, 910-923, Cairns, June 10-12.

Roten, D., K. B. Olsen, S. M. Day, Y. Cui and D. Faeh (2014), Expected seismic shaking in Los Angeles reduced by San Andreas fault zone plasticity, Geophysical Research Letters, 41, doi:10.1002/ 2014GL059411.

Roten, D., K. B. Olsen, Y. Cui, and S. M.Day (2015), Quantification of fault zone plasticity effects with spontaneous rupture simulations, to be submitted to Workshop on Best Practice in Physics-Based Fault Rupture Models for Seismic Hazard Assessment of Nuclear Installations, Vienna, Austria, Nov 18-20.

Shaw, J. H., A. Plesch, C. Tape, M. P. Suess. T. H. Jordan, G. Ely, E. Hauksson. J. Tromp, T. Tanimoto, R. Graves, K. Olsen, C. Nicholson, P. J. Maechling, C. Rivero, P. Lovely, C. M. Brankman, and J. Munster (2015), Unified Structural Representation of the southern California crust and upper mantle, Earth Planet. Sci. Lett., 415, 1-15, doi:10.1016/j.epsl.2015.01.016.

Taborda, R., and J. Bielak (2014),Ground-Motion Simulation and Validation of the 2008 Chino Hills, California, Earthquake Using Different Velocity Models. Bull. Seismol. Soc. Am., 104, 1876-1898, doi:10.1785/0120130266

Wang, F., and T. H. Jordan (2014), Comparison of probabilistic seismic hazard models using averaging-based factorization, Bull. Seismol. Soc. Am., 104, 1230-1257, doi: 10.1785/0120130263.

Withers, K.B., K.B. Olsen, S.M. Day (2015). Memory-efficient simulation of frequency dependent Q, Bull. Seismol. Soc. Am., in revision.

Zhou, J. (2014). Scalable Parallel Programming for High Performance Seismic Simulation on Petascale Heterogeneous Supercomputer, PhD Thesis, University of California at San Diego.

1