Help for Final Data Validation Procedures

In these steps, you will back-up the toolbox and have the option of invalidating any flagged data. Invalidation will exclude the data from AQS and future calculations. Depending on the qualifier flag that you have assigned to the data, you will decide whether to invalidate the data. You may also need to use your logbooks to determine the appropriate invalidation code to assign to the data. Please make sure that you have completed the “Logbook and External Information Data Flagging” steps before completing the steps outlined here. Only data that you have previously flagged will be eligible for invalidation. ITEP recommends that you complete these steps on a quarterly basis, however, you may do so more frequently as you see fit. As always, you can contact ITEP anytime if you have problems.

Before you begin the Final Data Validation Procedures in the toolbox, ITEP recommends that you make a back-up copy of the toolbox. ITEP recommends that you store all of your back-ups in an external location, for example on a network server, USB memory stick, etc. You should create a new folder in this location exclusively for your toolbox back-ups, if you have not already done so. Make sure to name this folder something obvious like ToolboxBackUp.

  1. Begin by closing the toolbox if it is open.
  2. Navigate to the folder where the toolbox is stored on your hard drive (C drive) and open it. For example, if you are using version 1.0 of the toolbox, you would open the TribalDataToolbox_v1.0 folder on your hard drive.
  3. Right click on the working version of the toolbox. If you are using version 1.0, the working version of the toolbox would be the “Tribal Data Toolbox_v1.0.mdb” file. Then click on the copy option.
  4. Navigate to the folder that you are going to use to store your back-ups in. Open this folder and from the Edit drop-down menu, choose Paste.
  5. Change the name of the file that you just pasted to something to indicate when you created the back-up file. For example, you could add the date that the file was created to the end of the name, such as “Tribal Data Toolbox_v1.0_091907.mdb” for a back-up file created on 9/19/2007.
  6. Now open the working version of the toolbox, for example “Tribal Data Toolbox_v1.0.mdb” if you are using version 1.0, on your hard drive (C drive) as you would normally do to work with the toolbox and follow the steps below.

From the Main Switchboard, click on the “Final Data Validation” button under the Quarterly Data Backup and Review category.

The following window will open.

The three boxes on this form represent the different types of data that you may have collected. You should complete the following steps for each type of data that you have imported into the Toolbox. The steps are similar for each data type.

  1. Make selections in all of the white fields with drop-down arrows for the data type you are going to review for invalidation. The fields vary depending on the data type.
  2. In the Date/Time and Date Range fields, you can begin typing the date to get closer to the value in the drop-down box and avoid having to scroll through all the date/time options. For example, if you wanted to select a start date and time of 8/20/2003 12:00 AM, you could begin by typing 8/20 and then clicking on the drop-down arrow. The following example illustrates how you would go about selecting ambient temperature data for the third quarter of 2003.

  1. After you have completed the selection of the data that you will be reviewing for invalidation, click on the “View All Qualified Data and Manually Add Flags to Invalidate” button.
  2. This button will bring up all of the data that you have previously flagged for the selections you specified in the white fields. (Note: if you have not previously flagged the data, the data will not show up when you click this button and therefore you will not have the option of invalidating the data.)
  3. Make sure that you have completed the “Logbook and External Information Data Flagging” steps to ensure that all of the data that should be qualified are qualified.
  4. The following screen shows the flagged data chosen in the previous step.

  1. All of the data in this example have a series of qualifier flags, including “OAT”. This means that the ambient temperature is out of range (Note: see the end of this help file for a complete list of qualifier flags assigned by the toolbox and their associated descriptions). Since the ambient temperature is clearly out of range at -999 degrees C, these data should be invalidated. This invalidation will exclude the data from calculations and AQS submittal. This needs to be done manually.
  2. Click in the InvalidationCode field to bring up a drop-down arrow.
  3. Click on this drop-down arrow for a list of Invalidation Codes and their associated descriptions.
  4. Scroll down the list until you find the one that best describes why you are invalidating the data value and click on it (although there may be multiple qualifier flags for each data value, only one Invalidation Code can be chosen per data value).
  5. You may need to consult your logbook to determine the reason the data value should be invalidated.
  6. If you cannot find a code to describe the reason for invalidating the data value, choose “AM” for Miscellaneous Void.
  7. If you have a series of data values that should all have the same Invalidation Code, you can copy the Invalidation Code in the above cell to the next cell by clicking in the destination cell and simultaneously pressing the <ctrl> key and the apostrophe ( ‘ ) key on your keyboard. If this becomes tedious because you have many data values that you need to invalidate with the same Invalidation Code, please contact ITEP for information on how to automate this task.
  8. Note that there may be cases when data have qualifier flags, but the data are still valid.
  9. Use your discretion in determining when to invalidate data based on your knowledge and logbook records.
  10. If you determine that the data are valid, the data should still be submitted to AQS and included in calculations. The InvalidationCode field should be left blank in these cases.
  11. Pay particular attention to PM10 and PM2.5 filter data with lab qualifier flags of “AX.” This means that the precision between the primary (RO) and collocated (CO) filter is out of range. This may be because one of the concentrations was low or high due to some issue such as a sampler motor burn out, etc. For example, the motor may have burned out on the primary sampler during a sampling period, but the collocated sampler did not have any problems during the same sampling period. In this case, you should invalidate the data from the RO filter, but you should not invalidate the data from the CO filter collected during the same sampling period, even though both records have a flag of AX.
  12. When you have finished reviewing all of the data values for invalidation, close the query window. If there are other pollutants or meteorological parameters that you need to review for invalidation, you must do so separately by completing Steps 1 through 5 of this document for each of those pollutants or meteorological parameters.

List of Qualifier Flags Assigned by the Toolbox

QualifierCode / QualifierDesc
AG / Filter Sample Time Not 24 Hours
AH / Filter Flow Rate and/or CV Out of Limits
AM / Filter Weighed More Than 30 Days After Sampling Period
AQ / Filter Retrieved More Than 7 Days After Placement Date
AR / Lab Miscalculation of Net Filter Weight
AS / Outdated Filter QC and/or Maintenance Check
AX / Out of Range Relative Percent Difference Between RO and CO Filter Samples
L1S / Out of Range Precision Point 1 QC Check
L1Z / Out of Range Zero QC Check
L2S / Out of Range Precision Point 2 QC Check
L3S / Out of Range Precision Point 3 QC Check
L4S / Out of Range Precision Point 4 QC Check
L5S / Out of Range Precision Point 5 QC Check
OAT / Out of Range Ambient Temperature
OBP / Out of Range Barometric Pressure
OIT / Out of Range Internal Temperature
OP / Out of Range Precipitation
ORH / Out of Range Relative Humidity
OSD / Out of Range Standard Deviation Wind Direction
OSR / Out of Range Solar Radiation
OWD / Out of Range Wind Direction
OWS / Out of Range Wind Speed
Q1 / Miscellaneous Qualifier (Assigned During Data Import)
Q2 / Miscellaneous Qualifier (Assigned During Review of Archived Data)
R / Out of Range Pollutant Concentration
STR / Out of Range Shelter Temperature
ZAT / Miscellaneous Ambient Temperature Qualifier
ZBP / Miscellaneous Barometric Pressure Qualifier
ZIT / Miscellaneous Internal Temperature Qualifier
ZP / Miscellaneous Precipitation Qualifier
ZRH / Miscellaneous Relative Humidity Qualifier
ZSD / Miscellaneous Standard Deviation Wind Direction Qualifier
ZSR / Miscellaneous Solar Radiation Qualifier
ZWD / Miscellaneous Wind Direction Qualifier
ZWS / Miscellaneous Wind Speed Qualifier

Revised 9/20/20071