The Evaluation of Red Flag Warnings

The Evaluation of Red Flag Warnings

in the Northwest Geographic Area

during the 2002 Fire Season

Executive Summary

Red Flag Warnings are issued by the National Weather Service (NWS) to notify wildland fire agencies of weather conditions that, in conjunction with critically dry fuels, could lead to a dramatic increase in fire danger or wildfire activity. Timely and accurate warnings enable wildland fire fighting agencies to manage critical resources and prepare appropriate suppression responses for protecting life and property. The combination of critical weather events (i.e. strong wind, low relative humidity and/or dry lightning) and low fuel moisture is defined as a “Red Flag” event. Fire Weather Watches are used to alert land management agencies in advance of possible Red Flag events.

The following summarizes the evaluation of Red Flag Warnings issued by the six NWS offices (Seattle, Spokane, Portland, Pendleton, Medford and Boise) within the Northwest Geographic Area during the 2002 fire season. The evaluation was performed in accordance with the Memorandum of Understanding (EXHIBIT C Forecast and Service Standards, Section E) between the Pacific Northwest Wildfire Coordinating Group (PNWCG) and the NWS. Red Flag Warnings were verified using lightning data, Remote Automatic Weather Station (RAWS) data, National Fire Danger Rating (NFDRS) indices, fuel moisture values, and fire information. In summary:

• Spokane, Seattle and Portland had False Alarm Rates (FAR) greater than .50, which means the majority of warnings issued by those offices did not verify. These offices had a tendency to “over-warn” for the situation.
• The majority of actual Red Flag events were correctly forecast, however the probability that these events are correctly forecast has worsened since 2000. There were a number of days in late July and August in which Red Flag Warnings should have been issued and weren’t for strong wind and low humidity in eastern Washington and eastern Oregon.
• There were 214 Red Flag Warnings issued by the NWS in the Northwest Geographic Area in 2002. This compares to 183 in 2000 and 100 in 2001. The increase in number this year can be attributed to the extended length of the fire season and the many long-duration fires in southern and eastern Oregon.
• Accurate and timely warnings were issued in advance of the “dry lightning bust” of July 12-13 and the strong northeast wind and low relative humidity in southwest Oregon July 27 through August 1.
• Procedural errors in issuing and canceling Red Flag Warnings continue. In one case, a Warning was issued by Medford for a non-existing fire weather zone.
• No problems were noted in the coordination of Red Flag Warnings among NWS offices this year. This was due to the creation of “seamless” red flag criteria among adjacent NWS offices and daily conference calls with NWCC meteorologists.
• The percentage of Warnings preceded by a Watch exceeded the 60% goal established in the PNWCG/NWS MOU.
• The Probability of Detection and False Alarm Rates in the Northwest in 2002 failed to meet the NWS Western Region averages published in 1999. Because 1999 was the last year of “dedicated” fire weather forecasters, this may be explained by the shift of fire weather duties to less experienced “core” forecasters.
• As “dedicated” fire weather meteorologists are replaced by “core” forecasters and “computer-worded” fire weather forecasts, a concerted effort must be adopted by NWS and NWCC meteorologists to monitor the current weather (i.e. radar, RAWS observations) and fuel conditions to ensure that red flag warnings are timely and accurate enhancing firefighter safety.

Introduction:

Red Flag Warnings are issued by the National Weather Service (NWS) to notify wildland fire agencies of weather conditions that, in conjunction with critically dry fuels, could lead to a dramatic increase in fire danger or wildfire activity. Timely and accurate warnings enable wildland fire fighting agencies to manage critical resources and prepare appropriate suppression responses for protecting life and property. The combination of critical weather events (i.e. strong wind, low relative humidity and/or dry lightning) and low fuel moisture is defined as a “Red Flag” event. Fire Weather Watches are used to alert land management agencies in advance of possible Red Flag events.

The following report is an evaluation of Red Flag Warnings issued by the six NWS offices (Seattle, Spokane, Portland, Pendleton, Medford and Boise) within the Northwest Geographic Area during the 2002 fire season (see appendix for a list of warnings by office and verification). This detailed evaluation was performed in accordance with the Memorandum of Understanding (EXHIBIT C Forecast and Service Standards, Section E) between the Pacific Northwest Wildfire Coordinating Group (PNWCG) and the NWS. Red Flag Warnings were verified using lightning data, Remote Automatic Weather Station (RAWS) data, National Fire Danger Rating (NFDRS) indices and fuel moisture values, and fire information. An examination of these Red Flag Warnings follows.

Findings:

1.False Alarm Rate (FAR)

The False Alarm Rate (FAR) is the percentage of Red Flag warnings that did not verify. The FAR can vary from 1.00 (no warnings correct) to .00 (all warnings correct). The more often an event is forecast and does not occur, the higher (worse) the score. Thus, it is a measure of Red Flag Warning accuracy. The closer the FAR is to .00, the more accurate the warnings. The False Alarm Rate for all warnings varied from .15 to .80, depending upon the office. Spokane, Seattle and Portland had False Alarm Rates greater than .50, which means that the majority of warnings issued by those offices did not verify. Improvement in FAR was noted in warnings issued by Boise and Medford compared to 2001.

Listed below are the dry lightning, wind/low RH and all warning False Alarm Rates for the six NWS offices.

FAR FAR FAR

Office Dry lightningWind/Low RH All Warnings

Spokane .68.57 .62

Seattle 1.00 .50 .80

Portland .37 .64 .55

Pendleton .28 .41 .36

Medford .31 .45 .42

Boise .50 .09 .15

2.Probability of Detection (POD)

The Probability of Detection (POD) is the percentage of actual Red Flag events that were correctly forecast. The more often an event is accurately forecast, the better the score. It can vary from 1.00 (all Red Flag events are correctly forecast) to .00 (all Red Flag events were not forecast). The Probability of Detection ranged from 1.00 at Seattle to .50 at Portland. In general, the majority of Red Flag events were correctly forecast. Since POD was not calculated in 2001, a comparison to the previous year is not possible. However, POD values in general have worsened since 2000.

Listed below are the dry lightning, wind/low RH and all warning Probability of Detection values for the six NWS offices.

POD POD POD

Office Dry lightningWind/Low RH All Warnings

Spokane .82.71 .75

Seattle N/A 1.00 1.00

Portland .83 .36 .50

Pendleton 1.00 .54 .67

Medford .58 .53 .54

Boise 1.00 .83 .85

3.Critical Success Index (CSI)

The Critical Success Index (CSI) is a combination of FAR and POD. It is the ratio of correct forecasts to the number of actual Red Flag events plus the number of incorrect forecasts. The best score is 1.00, the worst is 0. Critical Success Index values ranged from .73 at Boise to .20 at Seattle. Boise was the only office to have a higher CSI than the 1999 NWS Western Region average of .58.

Listed below are the dry lightning, wind/low RH and all warning Critical Success Index values for the six NWS offices.

CSI CSI CSI

Office Dry lightningWind/Low RH All Warnings

Spokane .30.36 .33

Seattle .00 .50 .20

Portland .56 .22 .31

Pendleton .72 .36 .48

Medford .49 .37 .39

Boise .50 .77 .73

4.Number of Red Flag Warnings

There were 214 Red Flag Warnings issued in the Northwest Geographic Area in 2002. This compares to 100 in 2001 and 183 in 2000. The increased number of Red Flag Warnings this year was due to the extended length of the fire season and the many large long-duration fires in southern and eastern Oregon. The number of warnings by office were as follows: Medford 68, Spokane 56, Pendleton 50, Portland 22, Boise 13 and Seattle 5.

5.Notable Red Flag Warning Successes

Accurate and timely Red Flag Warnings were issued well in advance of two significant events this year. The first event was the “dry lightning bust” of July 12-13. Watches and/or warnings were issued by all six NWS offices (Medford, Boise, Pendleton, Portland, Spokane and Seattle) prior to the lightning event. Between the 12th and 15th, the Northwest Geographic Area reported 375 new fires, including 16 large project-sized fires. The vast majority of these fires were located in southern and eastern Oregon. The Biscuit and Tiller Complex fires in southwestern Oregon continued to burn well into October.

The second event occurred in southwest Oregon between July 27th and August 1st. Strong northeast winds and extremely low relative humidity resulted in extreme fire behavior and significant increases in acreage burned on the Florence and Biscuit fires. Medford issued a Fire Weather Watch for these strong winds two days prior to the event on July 25th and upgraded to a Red Flag Warning July 27th. Red Flag Warnings remained in effect until August 1st. During this time, the Florence and Biscuit fires grew from approximately 11,000 acres to over 170,000 acres.

1

6.Coordination of Red Flag warnings/criteria among National Weather Service offices

No problems were noted with the coordination of Red Flag Warnings this year. This was due to better communication (daily telephone conference calls) between the Geographic Area Coordination Center and NWS meteorologists, and the creation of “seamless” red flag criteria among adjacent Weather Service offices with similar fire weather patterns, fuels and topography.

7.Missed Red Flag events

An evaluation of “missed” Red Flag events compared each office’s published Red Flag criteria on their web page to hourly RAWS data. The Predictive Services Branch of the Northwest Area Coordination Center daily archives 24-hour observations from approximately 200 RAWS. A data base query of each weather station produced a list of the hours during each day in which the criteria was either met or exceeded. Isolated occurrences were discarded; only those instances with multiple hours and stations were counted as “missed” warnings. The vast majority of missed warnings were due to strong wind and low relative humidity, and not dry lightning. There were a total of 67 missed warnings, most in eastern and southern Oregon.

1

D:/Verf_2002/rfw2002_eval.rtf

8.Procedural errors in the issuance of Red Flag Warnings

Procedural errors are those in conflict with National Weather Service (WSOM Chapter D-06, Western Region ROML W-10-96) or NFDRS directives. There were relatively few errors noted this year compared to past fire seasons. Most of the errors detected involved format, improper cancellation, or inconsistent statements between the general forecast (FWF) and the warning statement (RFW). Medford appeared to be responsible for the bulk of these procedural errors.

1. Percentage of Red Flag Warnings preceded by a Fire Weather Watch

The PNWCG and NWS Memorandum of Understanding (MOU) states in Exhibit C that, “at least 60% of all Red Flag Warnings will be preceded by a Fire Weather Watch.” This is to ensure that there is sufficient advance notification of Red Flag events to properly prepare firefighting resources. All six NWS offices achieved the 60% goal. The following lists by office the percentage of time warnings were preceded by watches: Portland 75%, Pendleton 62%, Medford 88%, Boise 100%, Seattle 100%, and Spokane 70%.

10. Coordination with Storm Prediction Center’s fire weather products

The NWS’s Storm Prediction Center (SPC) located in Norman, Oklahoma publishes daily Fire Weather Outlooks” on their web site. These Fire Weather Outlooks are forecasts of significant fire weather that could severely impact firefighting efforts on new or existing wildland fires. Depending upon the weather and fire danger, locations can be designated as: 1) “Critical Fire Weather Areas” (both graphical and text formats) or 2) if not as critical, mentioned only in text. If an area is not mentioned in the forecast, fire weather is not considered significant due to the weather or the fire danger level.

These Fire Weather Outlooks were compared to the watches and warnings issued by the six NWS offices in the Pacific Northwest to determine the consistency of forecasts produced by the NWS. SPC’s Fire Weather Outlooks were frequently in conflict with forecasts produced by the local NWS offices. There were nine days during the 2002 fire season when local offices had watches or warnings issued for the day (most times issued the previous day), yet SPC had no mention of significant fire weather in the Northwest Geographic Area. The most notable occurrence was July 13 when SPC forecast “wet lightning”, while most NWS offices had Red Flag Warnings for “dry lightning”. Inconsistency in SPC and local NWS office forecasts can result in confusion and lack of confidence in NWS products.

Appendix

Pendleton Red Flag Warnings

Date / Zones / Reason / Verification
June 26 / 631, 675 / Dry Lightning / Yes - 631 No - 675
July 07 / 632, 638 / Dry Lightning / Yes
July 10-11 / 630, 632, 633, 634, 635, 638 / Low RH, HI 6 / Yes
July 12 / 630, 632, 633, 634 / Dry Lightning / Yes - 632
No - 630, 633, 634
July 13 / 630, 631, 632, 633, 634, 635, 638 / Dry lightning / Yes (spotty rain but numerous large fire starts)
July 22 / 630, 632 / Dry lightning / Yes - 630 No - 632
July 24 / 631 / Dry lightning / Yes
Aug 01-02 / 630, 632, 633, 638 / Wind/Low RH / Yes – 630, 632, 638
No – 633

Missed 631

Aug 14 / 631, 675 / Wind/Low RH / Yes – 631
No – 675

Missed 630, 632, 635, 638

Aug 15-16 / 630, 631, 632, 633, 634, 635, 638, 675, 681 / Wind/ Low RH / Yes – 630, 631, 632, 638
No – 633, 634, 635, 675, 681
Sept 01 / 631 / Wind/Low RH / No
Sept 15 / 630, 632, 633, 634, 635, 638 / Wind/Low RH/HI 6/Psbl Dry Ltng / Yes – 630, 632, 635, 638
No – 633, 634
Sept 25 / 633, 634, 635, 638 / Wind/Low RH / Yes – 638
No – 633, 634, 635
Aug 10 / Wind/Low RH / Missed 631
Aug 17 / Wind/Low RH / Missed 630, 631, 632, 635, 638, 681
Aug 30 / Wind/Low RH / Missed 631
Sept 14 / Wind/Low RH / Missed 630, 633, 638

Total Warnings: 50Dry Lightning: 18Wind/low RH: 32

Number of Events: 13Events Preceded with a Watch: 8 or 62%

Missed Warnings: 16Dry Lightning: 0Wind/low RH: 16

False Alarm Rate:Dry Lightning . 28Wind/Low RH . 41All . 36

Probability of Detection:Dry Lightning 1.00Wind/Low RH . 54All . 67

Critical Success Index:Dry Lightning . 72Wind/Low RH . 36All . 48

Note:For highest accuracy, False Alarm Rate should approach .00 and Critical Success Index and Probability of Detection 1.00

Calculations:a = correct warnings

b = incorrect warnings

c = missed warnings

False Alarm Rate: All 1 – (a/a + b) = 1 – (32/32 +18) = 1 – (32/50) = 1 - .64= . 36

Dry Ltng1 – (a/a + b) = 1 – ( 13/13 +5) = 1 – (13/18) = 1 - .72 = . 28

Wind/Low RH1 – (a/a + b) = 1 – (19/19 +13) = 1 – (19/32) = 1 - .59 = . 41

Probability of Detection:Alla/a + c = 32/32 + 16 = 32/48 = . 67

Dry Ltnga/a + c = 13/13 + 0 = 13/13 = 1. 00

Wind/Low RHa/a + c = 19/19 + 16 = 19/35 = . 54

Critical Success Index:Alla/a + b + c = 32/32 + 18 + 16 = 32/66 = . 48

Dry Ltnga/a + b + c = 13/13 + 5 + 0 = 13/18 = . 72

Wind/Low RHa/a + b + c = 19/19 + 18 + 16 = 19/53 = . 36
Spokane Red Flag Warnings

Date / Zones / Reason / Verification
June 26 / 676, 677, 680, 682, 684, 685, 687 / Low RH/wind/Dry Ltng / No (no strikes except isold 687)
July 03-04 / 676, 677, 684 / Low RH/wind / Yes – 684
No – 676, 677 (Canx 2100 PDT 7-03)
July 07 / 676, 677, 680, 682, 684, 685, 687 / Dry Lightning / No (ltng was wet .25-.50 inches)
July 12 / 673, 676, 677, 680, 682, 684, 685, 686, 687 / Low RH/ HI 6 / Yes
July 13 / 686 / Low RH/HI 6 / Yes
July 13 / 673, 676, 677, 680, 682, 684, 685, 686, 687 / Dry Lightning / No – 676, 677, 680
Yes – 673, 682, 684, 685, 686, 687
July 23 / 676, 677, 680, 682, 685 / Dry Lightning / No – 676, 677 (no strikes)
Yes – 680, 682, 685

Missed – 686, 687

Aug 14 / 673, 677, 684, 686, 687 / Wind and low RH / No (only 1 station 673, 680, 684, 686)
Aug 15-16 / 673, 676, 677, 680, 682, 684, 685, 686, 687 / Wind and low RH / Yes – 684
No – 673, 676, 677, 680, 682, 685, 686, 687 (only 1 station 673, 680, 685)
Sept 15 / 686 / Wind/Low RH/HI 6 / No

Missed - 684

July 27 / Wind/Low RH / Missed - 684
July 30 / Wind/Low RH / Missed - 684
Aug 17 / Wind/Low RH / Missed – 673, 684

Total Warnings: 56Dry Lightning: 28Wind/low RH: 28

Number of Events: 10Events Preceded with a Watch: 7 or 70%

Missed warnings: 7Dry Lightning: 2Wind/low RH: 5

False Alarm Rate:Dry Lightning . 68Wind/Low RH . 57All . 62

Probability of Detection:Dry Lightning . 82Wind/Low RH . 71All . 75

Critical Success Index:Dry Lightning . 30Wind/Low RH . 36All . 33

Note:For highest accuracy, False Alarm Rate should approach .00 and Critical Success Index and Probability of Detection 1.00

Calculations:a = correct warnings

b = incorrect warnings

c = missed warnings

False Alarm Rate: All 1 – (a/a + b) = 1 – (21/21 +35) = 1 – (21/56) = 1 - .38= . 62

Dry Ltng1 – (a/a + b) = 1 – ( 9/9 + 19) = 1 – (9/28) = 1 - .32 = . 68

Wind/Low RH1 – (a/a + b) = 1 – (12/12 +16) = 1 – (12/28) = 1 - .43 = . 57

Probability of Detection:Alla/a + c = 21/21 + 7 = 21/28 = . 75

Dry Ltnga/a + c = 9/9 + 2 = 9/11 = . 82

Wind/Low RHa/a + c = 12/12 + 5 = 12/17 = . 71

Critical Success Index:Alla/a + b + c = 21/21 + 35 + 7 = 21/63 = . 33

Dry Ltnga/a + b + c = 9/9 + 19 + 2 = 9/30 = . 30

Wind/Low RHa/a + b + c = 12/12 + 16 + 5 = 12/33 = . 36
Medford Red Flag Warnings

Date / Zones / Reason / Verification
July 12 / 620, 621, 622, 623, 624, 625 / Dry Lightning (issued 7-11),
Dropped zones 620, 622 on
7-12 am fwf / Yes – 623, 624, 625
No - 621
July 12 / 617, 620, 622 / Dry lightning / Yes – 617, 620, 622
Missed – 616
Missed (7/13) – 616, 619, 620, 621, 622, 623, 624
July 22 / 616, 617, 619, 620, 621, 622, 623, 624, 625 / Dry lightning / Yes – 617, 620, 621, 622, 623,
No – 616, 619, 624, 625 (wet 624, 625)
July 27 / 617, 618, 619, 620, 621, 624, 625 / Wind/Low RH / Yes – 618, 625
No – 617, 619, 620, 621, 624
July 28-29 / 618, 619, 620, 621 / Wind/Low RH / Yes – 618, 620
No – 619, 621
July 30-31 Aug 01 / 618, 619, 620, 621 / Wind/Low RH
Dropped 620, 621
on 7-31 pm fwf / Yes- 618, 619, 620
No – 621

Missed 616

Aug 07-10 / 618, 619, 620, 621 / Wind/Low RH
Canx 1145 am 8-10 / Yes – 618, 619, 620
No – 621

Missed 623

Aug 26-28 / 618, 619, 620, 621 / Wind/Low RH
Dropped zone 621 on 8-26 pm fwf and rfw / Yes – 618
No – 619, 620, 621
Aug 31 / 618, 619, 620 / Wind/Low RH / Yes – 618, 619
No – 620
Sept 01-02 / 618, 619, 620 / Wind/Low RH / Yes – 618, 619
No – 620
Sept 14 / 624, 625 / Wind/Low RH / Yes – 625

No – 624

Sept 18 - 21 / 618, 619 / Wind/Low RH / Yes – 618, 619 (Warning issued too early due to 2 inch rains the 17th)
Sept 23 - 26 / 618, 619 / Wind/Low RH / Yes – 618, 619

Missed 623

Sept 24 - 25 / 620 / Wind/Low RH / Yes
Oct 05-09 / 618, 619 / Wind/Low RH / Yes
Oct 11-12 / 617, 618, 619, 620, 621, 622, 623 / Wind/Low RH / No (wind/low RH event occurred
Oct 14-16)
Oct 31-Nov 01 / 616, 618, 619, 620, 621 / Wind/Low RH / Yes
Aug 12 / Wind/Low RH / Missed 619, 620
Aug 13 / Wind/Low RH / Missed 616, 619, 620, 623
Aug 14 / Wind/Low RH / Missed 625
Aug 17 / Wind/Low RH / Missed 625
Aug 18 / Wind/Low RH / Missed 620
Sept 3 / Wind/Low RH / Missed 625
Sept 4 / Wind/Low RH / Missed 625
Sept 10 / Wind/Low RH / Missed 620, 623
Oct 05-08 / Wind/Low RH / Missed 620, 623
Oct 14-16 / Wind/Low RH / Missed 616, 618, 619, 620, 623
Oct 31-Nov 01 / Wind/Low RH / Missed 623

Total Warnings: 68Dry Lightning: 18Wind/low RH: 50

Number of Events: 16Events Preceded with a Watch: 14 or 88%

Missed warnings: 32Dry Lightning: 8Wind/low RH: 24

False Alarm Rate:Dry Lightning . 31Wind/low RH . 45All . 42

Probability of Detection:Dry Lightning . 58Wind/low RH . 53All . 54

Critical Success Index:Dry Lightning . 49Wind/low RH . 37All . 39

Note:For highest accuracy, False Alarm Rate should approach .00 and Critical Success Index and Probability of Detection 1.00

Calculations:a = correct warnings