CEDAR-GEM Modeling Challenge Workshop

Conveners: Masha Kuznetsova, Ja Soon Shim, Barbara Emery, Aaron Ridley, Delores Knipp, Naomi Maruyama, Tim Fuller-Rowell, Tim Guild, Jan Sojka, Geoff Crowley

The Metrics and Validation Focus Group together with the CEDAR modeling community organized the Joint CEDAR-GEM Modeling Challenge Workshop. The CEDAR-GEM Challenge is build upon GEM GGCM and CEDAR ETI Challenges. During the Workshop the GEM and CEDAR communities shared the experience and lessons learned from the first rounds of Challenges, addressed topics of common interest and initiated joint model validation projects focusing on effects of geospace model coupling on metrics results. The Joint CEDAR-GEM Challenge Workshop was well attended by modelers, data providers and users of space weather models.

Both CEDAR and GEM communities have recognized that due to the maturity and increasing complexity of state-of-the-art space weather models, there is a great need for a systematic and quantitative evaluation of different modeling approaches. During the last two years both GEM and CEDAR communities addressed this need by organizing and implementing comprehensive, community-wide efforts to test model predictions against observations. In the summer of 2008 the GEM GGCM Metrics and Validation Focus Group initiated a series of metrics studies (aka GEM 2008Modeling Challenge) focusing on the inner magnetospheric dynamics and ground magnetic field perturbations. A year later the CEDAR community initiated the IT modeling challenge called CEDAR Electrodynamics Thermosphere Ionosphere (ETI) Challenge. The goal of the two Challenges is to evaluate the current state of the space physics modeling capability, to facilitate interaction between research and operation communities in developing metrics for space weather models, to address challenges of model-data comparison, to track model improvements over time, to facilitate collaboration among modelers, data providers and research communities, and provide feedback for further model improvement. The Community Coordinating Modeling Center (CCMC) is supporting GEM, CEDAR and Joint Challenges and maintaining a web site with interactive access to model output archive and observational data used for metrics studies.

The Workshop had three breakout sessions. One session (2 hours Tuesday, June 28 1:30-3:30 PM) focused on climatology projects (time periods longer that 3 months). The first hour was dedicated to presentations from CEDAR community that has been performing climatological validation of ionosphere thermosphere models for years. Presentations during the second hour demonstrated that GEM community is getting increasingly interested in performing climatological validation of component GGCM models.

Katie Garcia from Boston University talked about her 2007 paper of using a long MHD model run of real solar wind inputs to statistically characterize the magnetopause standoff distance in the MHD model over a variety of solar wind conditions. Mike Liemohn and Roxanne Katus (University of Michigan) presented initial results of very long (several years) continuous HEIDI simulations of the ring current. Lutz Rastaetter and Hyesook Lee (CCMC) demonstrated model output archive from CCMC real-time simulations and presented examples of how this archive can be used for climatology projects. CCMC also demonstrated the newly developed interactive analysis tool applicable to analysis of long time series. The decision has been made to arrange a session on climatology projects during the 2012 GEM Summer Workshop. The session will be organized by Tim Guild and Lutz Rastaetter.

Two sessions (4 hours Thursday June 30, 10:00 am -noon, 1:30-3:30 PM) focused on less than 3 days long event:

E.2006.348: 2006/12/14 (doy 348) 12:00 UT - 12/16 00:00 UT

E.2001.243: 2001/08/31 (doy 243) 00:00 UT - 09/01 00:00 UT

E.2005.243: 2005/08/31 (doy 243) 10:00 UT - 09/01 12:00 UT

E.2005.135: 2005/05/15 (doy 135) 00:00 UT - 05/16 00:00 UT

E.2005.190: 2005/07/09 (doy 190) 00:00 UT - 07/12 00:00 UT

E.2003.302: 2003/10/29 (doy 302) 06:00 UT - 10/30 06:00 UT (optional)

List of physical parameters to be used for metrics studies:

Ionosphere/Thermosphere models or coupled model components:

  • Vertical and horizontal drifts at Jicamarca
  • Neutral density at CHAMP orbit (Nden)
  • Electron density at CHAMP orbit (Eden)
  • NmF2 from LEO satellites (CHAMP and COSMIC) and Incoherent Scatter Radars (ISRs)
  • HmF2 from LEO satellites (CHAMP and COSMIC) and ISRs
  • Temperature Tn and neutral winds obtained by Fabry-Perot Interferometer at 250 km (Arrival Heights, Antarctica; Resolute Bay, Canada)
  • Ne, Te, Ti at 300 km (Millstone Hill, Sondrestrom, EISCAT, Svalbard ISRs).
  • Ion vertical velocity at Sondrestrom ISR

Geospace models or coupled model components:

  • Magnetic field at geosynchronous orbit
  • Ground magnetic perturbations
  • Dst index
  • Auroral oval position (high latitude boundary)
  • Auroral oval position (low latitude boundary)

Parameters along DMSP tracks:

  • Poynting flux (Joule heating) into ionosphere along DMSP tracks
  • Plasma Velocity (Vx - along track, Vy cross track, Vz - vertical)

Additional time series in support of simulation results analysis:

  • Cross polar cap potential (northern and southern hemisphere)
  • Joule heating (or Poynting flux) integrated over each hemisphere in GW.

Antti Pulkkinen and Ja Soon Shim presented reviews of the first round of GEM and CEDAR Challenges results. Antti demonstrated the progress in ground magnetic perturbation Challenge from the first GEM metrics study to the operational geospace model selection. The first round of GEM GGCM Modeling Challenge has so far resulted in three publications (with all Challenge participants as co-authors). This ground magnetic perturbation metrics study is being used as a foundation for the independent model validation activity conducted by CCMC in support of NOAA SWPC operational geospace model selection. The presentation triggered a lively discussion on operational model selection process. Howard Singer representing NOAA SWPC expressed interest in community feedback on the selection process of an operational model. Modelers and model users are encouraged to send their comments and ideas on model selection processes to CCMC no later than October 1st, 2011. Ja Soon Shim presented a summary of the first CEDAR metrics studies. It was demonstrated that model performance varies from event to event. None of the model ranks at the top for all used metrics. Empirical models ranked high on the average or during the quite times, while physics based models better represent dynamics. The community agreed that there is sufficient material for two papers to be submitted to Space Weather Journal before the Fall AGU. Ja Soon will prepare paper drafts and send them to all co-authors by October 1st, 2011. Results of the first round of Challenges will be used as a baseline for future studies.

Dan Welling and Lutz Rastaetter summarized the results of the Dst index challenge (joint effort of Metrics and Validation and Inner Magnetosphere Focus Groups with CEDAR community participation). Interest to the Dst challenge was demonstrated by a broad participation with more than 30 model submissions (including global magnetosphere models, ring current models, statistical models, and real-time data analysis). Lutz presented a demo of the updated CCMC interface that instantly calculates skill scores for selected model settings and different metrics types (prediction yield, cross-correlation and timing error). Metrics results are presented bytwo-dimensional diagrams that combine correlation with prediction efficiency or prediction yield with timing error. Statistical specifications were shown toperform better than mostphysics-based models. Among physics-based models the best results are produced by models with self-consistent global MHD - ring current coupling. Inconsistency in USGS and Kyoto Dst (8 nT offset) for some of the events was found. USGC participant (Jennifer Gannon) is requested to clarify the issue. Lutz Rastaetter will prepare a paper draft by October 1st with the goal to submit it to the Space Weather Journal before the end of 2011.

Delores Knipp introduced a new metrics study that involves comparison of DMSP measurements of Poynting flux into ionosphere with ionosphere Joule heating that can be produced by geospace and ionosphere models. Poynting Flux/Joule Heating Challenge involves both GEM and CEDAR communities. Lutz Rastaetter showed very preliminary analysis of the first model result submissions. First comparisons of half-orbit integrated time series look promising. The discussion will be continued at the mini-workshop in San Francisco. More modelers expressed interest in participating in the Challenge. To be included in summary reports planned for the mini-workshop and Fall AGU modelers should submit their results (Joule heating along DMSP orbits) prior to November 1st, 2011.

Aaron Ridley led the discussion on IT/geospace model coupling. For the first project it was suggested to study the role of drivers on ionosphere model results. Examples of drivers that provide ionosphere potential pattern include: Weimer, AMIE, Hardy, MHD output, RCM output. Aaron offered to share the F90 library that allowed him to easily switch between different drivers. The library takes solar wind data, Hemispheric Power, Kp, time, magnetic latitude and magnetic local time and return potential, average energy or energy flux. To proceed with the project Aaron agreed to make libraries, instructions on how to use them and necessary data files available for download via CCMC Web site. CCMC will work on converting ionosphere electrodynamics model outputs to platform independent and self-descriptive formats (cdf, netcdf, hdf5) that allow direct access to both the model data as well as the embedded metadata using CCMC Kameleon access and interpolation library. All tools should be available prior to October 1st 2011 so modelers can run simulations with different drivers (Weimer 2005 and AMIE as a first priority) and submit results before November 1st, 2011. Events to begin with are: E.2006.348 (observations for maximum ionosphere parameters are available) and E.2005.243 (priority event for the Poynting Flux/Joule heating study). The experience will be discussed at the mini-workshop.

The last hour of the workshop was dedicated to challenges of model-data comparison studies and how to address them.Robert Schunk discussed problems with physics-based models that can lead to uncertainties in model output.Issues include simplified math formulation, uncertain input parameters, incomplete or approximate coupling, insufficient spatial and temporal resolution and missing physics. One suggested solution is to conduct two simulations - one with lower end and one with upper end of uncertainties. Spread in output provides an estimate of the uncertainty. Uncertainty parameters should be identified for each model participating in the metrics studies. Current CCMC model results submission interface and model results archive allowing multiple submissions for the same model with different model settings facilitate the uncertainty analysis. It was agreed that the uncertainty analysis and different approaches to ensemble modeling is an important topic that should be addressed at future workshops.

Yihua Zheng introduced a new auroral oval boundaries metrics study that is of special interest to a number of space weather model users (including US Air Force). Yihua addressed the challenging issues in auroral oval metrics studies: How to define the equatorward boundary of the auroral oval from simulations? How to do model-data comparison and measure the model performance? Several methods based on threshold in particle precipitation fluxes were introduced. Ionosphere Joule heating pattern can also be used for models that do not include ring currents. Modelers are requested to submit poleward and equatorward boundaries locations with 1 hour local time resolution (24 points for each boundary for each time step). The first priority event is E.2005.243. Results will be discussed at the mini-workshop. Model output submission deadline is November 1st 2011.

Barbara Emery, Larisa Goncharenko and Anthea Coster addressed challenges of metrics studies for global time dependent observational data sets (Total Electron Content used as an example). Examples presented by Larisa Goncharenko demonstrated that longitude slices can capture many storm features. To make the TEC study manageable it was agreed to choose up to 5 longitude slices (5 deg in glon and 5 deg in glat with 15 - 36 lat bins) that corresponds to about 75 stations. Larisa and Anthea will provide TEC data files to CCMC for the E.2006.348 priori to September 15th, 2011. CCMC will post model output format description and add TEC to the submission interface by October 1st, 2011. Modelers will submit model output in required format by November 1st, 2011. The first results will be presented at the Fall AGU.

The CEDAR community expressed interest to continue working with GEM on joint model validation projects. To facilitate the continuation of GEM-CEDAR collaboration, members of both communities expressed interest in arranging a GEM-CEDAR modeling session during the GEM mini-workshop in San Francisco.

Presentations from the workshop, CCMC metrics tools, instructions on how to participate, and action plan summary with deadlines can be found at the Challenge Website: