IceCube Project Monthly Report - October 2009

Accomplishments

  • October was one of the most smooth and trouble-free data taking months this year. The overall detector uptime was 99.0% and the clean in-ice uptime was 95.8%
  • An improved program for generating simulated detector data was released in October and is being tested to ensure compatibility with previously generated simulation data.
  • IceCube personnel started traveling to the South Pole in October. The first drillers and the two detector operators that will winter at South Pole in 2010 are scheduled to arrive at Pole the first week in November 2009.
  • Three undergraduate students began atrip to the South Pole as part of the Upward Bound program that encourages science experiences for students from under represented populations. The students travel on the Swedish Icebreaker Oden with a special IceTop tank outfitted with DOMs to calibrate the energy response of the tanks to solar particles.

Cost and Schedule Performance – The project is 93.4% complete. Remaining contingency is $6.8 million. There has been no change to the NSF MREFC funding requirements of $242.1 million since the project performance baseline was established in February2004.

The Schedule Variance at the end of September 2009 was -$8.2k. The cost variance at the end of September 2009 was $2,131k (compared to $1,844 last month). This favorable variance is principally due to lower labor and on-ice support costs for Raytheon Polar Services Corporation and the Air National Guard in FY2009. In addition, training costs for drilling were less than budgeted (Implementation) and there are still several lagging invoices for cables (Instrumentation) and computing infrastructure (Pre Operations).

The baseline schedule for string installation is 18 strings in December 2009 – January 2010 and 3 strings in the final season, December 2010 – January 2011. A total of 19 strings were installed last season. UW and the IceCube collaboration will propose installing six additional strings to complete the 80-string array originally proposed plus the DeepCore.

Resources are in place for the successful completion of the MREFC project in FY2011.

UW-Madison and the IceCube collaboration submitted revised budgets for M&O for FY2011-2015. The revisions were made in response to guidance from the NSF and the IceCube International Oversight & Finance Group (IOFG). Included in the submittal were responses to the recommendations from the NSF panel that reviewed the original proposal.

Contingency Status and Plans ─Two change requestswere implemented in the past month: IC0157 Additional computer science support by NPX Designs and IC0159 surface-to-DOM cable assembly at Sea Con. The available Contingency is now $6,847M.

Change Log – IceCube Total Project Budget Baseline ($K)
No. / Description / Date Approved / Total Baseline / Allocated Budget / Allocated Budget Change / Contingency Budget / Budgeted Cost of Work Remaining
CR149 / UA Graduate Student / 07/13/09 / 275,503 / 268,095 / 39 / 7,408
CR150 / Rebaseline RPSC FY10 budget / 07/13/09 / 275,503 / 267,808 / -287 / 7,696
CR151 / Add Capital Equipment to 1.9.4 / 07/27/09 / 275,503 / 267,908 / 100 / 7,596
NA / Status as of June 2009 / 275,503 / 267,908 / 0 / 7,596 / 20,069
NA / Status as of July 2009 / 275,503 / 267,908 / 0 / 7,596 / 19,036
CR154 / ICL Rooftop access / 09/29/09 / 275,503 / 267,908 / 0 / 7,596
CR155 / Data storage upgrade / 09/30/09 / 275,503 / 268,371 / 464 / 7,132
CR156 / SUBR distributed computing / 09/28/09 / 275,503 / 268,409 / 37 / 7,095
NA / Status as of August 2009 / 275,503 / 268,409 / 0 / 7,095 / 18,541
CR157 / AdditionalSupport for Monitoring South Pole Computing Systems / 10/29/09 / 275,503 / 268,516 / 107 / 6,988
CR159 / Remaining Surface-to-DOM cables assembly at Sea Con / 11/03/09 / 275,503 / 268,657 / 141 / 6,847
NA / Status as of September 2009 / 275,503 / 268,657 / 0 / 6,847 / 17,837
Risk Assessment & Potential Contingency Adjustments
1. / Contingency potentially required for technical, cost and schedule risks associated with the approved scope of work. Risk assessments are made at WBS-Level 4 to determine the value of the risk exposure as a percent of the cost of work remaining. / $1,547K
2. / Pre-Operations costs for additional capacity to the data storage and network systems both at the South Pole and in the North, and for extending software development efforts. / $2,100K
3. / RPSC Estimated FY09 Closeout (-$700K) and moving the support for one string from FY09 to FY11 (+$250K) / -$450K
4. / RPSC estimate of base cost to support the installation of six strings during the 2010/11 drilling season. The current RPSC baseline already includes support for 80 strings. / $1,340K
5. / The cost to retro IceCube equipment/materials from the South Pole at the end of the project in FY2012. This is a conservative estimate based on Rev. 8 of the RPSC budget. / $1,050K
6. / Potential cost to retain experienced key personnel in the final season to ensure the success of the last drilling and installation season. / $500K
7. / Potential additional Fuel Cost exposure during the last season of drilling (FY2011) / $450K
Total / $6,537K
Available Contingency as of / Sep 30, 2009 / $6,847K

Drilling and Installation –

South Pole Activities:

  • Season Equipment Site drift surveying was completed and excavation begun.
  • First wave of drillers scheduled to arrive on-station early November.
  • Hose Heating System monitoring continues – no issues.
  • Do Not Freeze Modules (Drill Control Center & Tower Operations Structures 1-2) power/heat monitoring continues – no issues.

Off-Ice Drilling Activities

  • Staffing: All driller positions are filled (w/alternates). One driller failed physical qualification (NPQ’d) and an alternate was secured.
  • Staffing: Full-time and seasonal drillers began deploying, with the 1st wave scheduled to arrive at South Pole Nov 4th.
  • Schedule: The Integrated Master Schedule and Master Site Plans are complete – both reflecting a 20-hole plan, w/19 surface cable trenches, and 14 IceTop Pits.
  • Cargo: The last of planned air and vessel shipments to Pole concluded in Oct.
  • Materials: Retrograde materials for 10/11 season were procured and shipped via vessel to McMurdo for pre-staging.
  • The Northern Hemisphere Support Group was re-established with 24/7 support numbers and contacts to expeditiously provide support for project requests originating from South Pole.
  • Procedure (SOP) updates were completed in October.
  • Hazard Analysis updates were completed.
  • Development of the EHWD User Manual (Turn-Over Package) continued through October. Script sequences are complete and ready for video shooting at NPX. Drawings and documentation updates/collection continued through the month. Related activities for South Pole are identified and planned.

Off-Ice Installation Activities

  • Staffing: The 4 primary deployer positions are filled. Two are still pursuing physical qualification (PQ), but are expected.
  • All equipment is shipped; Schedules and staffing plans are in place; Procedures and hazard analyses are updated.
  • The installation team will begin to arrive at the Pole around November 23rd.
  • A list of Special Devices is developed. Weekly status meetings continue to confirm most of these sub-projects will meet ship dates and be ready for installation at North Pole Station.

Detector Commissioning and Verification - Verification continues to run smoothly. Modified versions of the Verification scripts for the new operating mode for the Processing and Filtering (PnF) were prepared and are ready for testing on the South Pole Test System.

No additional progress to report on tests of waveforms into the standard processing.

Calibration and Monitoring – Single-LED flasher runs continue to be taken to aid in the study of ice properties.

As per the discussion, Standard Candle laser data was taken for energy calibration of the IceCube 59 string configuration. These data are being analyzed.

Data Acquisition Hardware & Software – Three software release candidates (“El Farolito”) were issued in October 2009. In each case, minor bugs were uncovered when the release candidate was deployed for the 24-hour test mode at South Pole. The bugs were caught within the first hour of the run so that the eventual impact on clean detector live time was less than 0.5%. At the end of October the IceCube DAQ is still running the Delfina software release from July 2009.

Sensitive to even this small deduction from the data quality of IceCube, we are continually trying to develop better ways to utilize the South Pole Testing System (SPTS) in the north to catch DAQ bugs before the software is implemented on the South Pole System (SPS). This is an on-going endeavor.

Detector Up-Time: 99.0%

IceCube (in-ice) clean runs Up-Time: 95.8%

Unscheduled Downtime: 0.2%

Definition of the terms:

“Detector Up-Time” is the percentage of the time period for which the pDAQ data acquisition was acquiring data and delivering at least 500Hz of event rate. This uptime measure therefore includes periods in which the detector was taking data with a partial detector enabled or with light contamination from calibration sources.

“Clean run Up-Time(s)” is the percentage of the time period considered to have pristine data (standard hardware and software configurations) with the full nominal detector enabled, not contaminated with light from calibration sources and for which no serious alerts were generated by the monitoring, verification or other. The criteria applied are not algorithmic but rather represent the Run Coordinator’s overall impression of the quality (including uniformity) of the runs/data.

October was one of the most smooth and trouble-free data taking months this year. The overall detector uptime was 99.0% and the clean in-ice uptime was 95.8%.

The few normal occurrences of equipment/systems malfunctions (e.g DOR cards giving bad data, or a DOM hub crashing or hanging up) were promptly alerted by the automatic monitoring systems and swiftly corrected by Winterover intervention.

Successful special runs for ice properties using flashers were taken. A special trigger test run was taken to study trigger configurations for next-year to capture data related to slow moving particles.

Data Handling - South Pole systems in the IceCube Laboratory began the 59-string configuration physics operation on May 20th and it continues to run smoothly. The figure below shows the daily satellite data transfer rates in gigabytes per day for October 2009. The purple is filtered physics data and the green is calibration and test data.

Information technology work focused in two areas: planning for South Pole systems upgrades to handle the additional strings and data rate in 2009-2010 installation season, and upgrades to data warehouse and offline processing infrastructure.

In preparation for pole, a detailed plan for South Pole system upgrade and maintenance was put into place. Work included an upgrade computer systems and the IceCube Lab to support the increased data volume from 77 strings. Additionally, major upgrades to the P&F filtering system and the taping system took place. Changes to the taping system will now allow for two taped copies of data to be made.

In the northern processing centers, two changes to the data processing machines will cause significant increases in analyzing data coming off the detector. Commissioning of a new high performance-computing cluster with 512 cores effectively tripling the processing speed. The new processing power will eliminate a significant delay in processing of 40-string configuration data, which has backed up 59-string configuration data (run completed in May 09) and the current 77-string configuration data processing. Expansion of the data warehouse by 234 TByte will also improve processing speeds.

Online & Offline Filtering, Software & Database - The online filtering system for the 59-string run, which began in May, continues smoothly with daily satellite transmission of filtered data to the northern data warehouse at the University of Wisconsin. Offline Level 1 and Level 2 processing(L1 and L2) processing in the northis paced by the availability of resources, both personnel supported via the Core M&O award and In-Kind service labor distributed across the collaboration.

As predicted, IceTray version 3 is in full release, with a follow up release containing minor additions to support simulation already prepared. Preliminary support for the conversion to the snow leopard operating system is now in place. Preparation for the start of IC77 filtering is underway, with a planned release for deployment to Pole following test runs.

L1/L2 processing of the 40-string configuration run that ended in April 2008 is nearing completion. The chart below illustrates that there is only about two months of the total of 13 months of remaining to be processed. Processing of the 59-string data run that ended in May 2009 is being prepared and will begin in November thanks to the processing upgrades.

The IceCube 40 string configuration processing status is indicated in the figure below.

Simulation - Improvements continue to be made in the simulation programs. During October, a new simulation release candidate was made available. Production tests are currently running to verify if the current release is compatible with previously produced data and to determine if new features can be used to improve coincident muon simulation.

The simulation production plan for the next 6-12 months is under preparation. Currently, estimates of CPU time and data size are being established, as well as a plan to distribute jobs across processing sites.

The figure below shows the global simulation production runtime for the last month:

Education and Outreach –Several undergraduate students began their trip to the South Pole as part of the Upward Bound program, which encourages science experiences for students from underrepresented populations. Two University of Wisconsin River Falls (UWRF) students and one student from UW Rock County will travel on the Oden Cruise ship from Sweden to Antarctica. They are traveling with a special IceTop tank outfitted with four DOMs to calibrate the energy response of the tanks to solar particles. The trip will be completed in three legs with Drew Anderson (UWRF) traveling on the first leg from Sweden to Montevideo, Uruguay. Samantha Jakel (UW-Rock County) will take over and accompany the tank to McMurdo Station. Finally, Kyle Jero (UWRF) will monitor the tank from McMurdo to Punta Arenas, Chile. The students will blog about their experiences at:

UW-Madison is planning an outreach event aimed at a Spanish-speaking audience. Explorando las Ciencias will be on November 15th and will include hands-on science activities. In addition to planning and organizing the event, three native Spanish speakers who are part of the IceCube research team will be at the event to answer questions and interact with visitors.

Quality Assurance and Safety – As deployment begins, the safety team will review procedures and check structures and processes at the Pole. A major accomplishment this month was the creation of Enhanced Hot Water Drill (EHWD) checklists. They were printed poster size for display in the Tower Operations Structure (TOS) and MECC structures at the Pole. The following nine posters were sent to the Pole:

  • Weight Stack Assembly
  • Updated EHWD Hose Inspection Criteria
  • EHWD Hose Section Change-Out
  • EHWD Hose reel Move
  • TOS Pre-Drill Set-Up
  • Tower/TOS Move
  • Heated Hose Specification Poster
  • EHWD/SES Camp Water Flow Diagram
  • IceCube Lock-Out/Tag-Out Policy

Monthly reports are posted at IceCube Monthly Reports.

1