Comparison of Alternative Measurement Methods

Comparison of alternative measurement methods

1. Introduction

1.1.This document describes the comparison of the accuracy (trueness and precision) of an analytical method with a reference method. It is based on ISO-5725-6:1994(E), section 8 (Comparison of alternative methods). Where possible the ISO text was taken over and terms used in this document are in accordance with ISO definitions. However the ISO text differs on the following main points from the present text:

1.2.In the ISO standard the reference method is an international standard method that was studied in an interlaboratory fashion. This means that the precision (2) is known. Here we consider the situation in which a laboratory has developed a first method (method A) and validated this, and later on wishes to compare a new method (method B) to the older, already validated method. The former will be referred to as the reference method. Only an estimate of the precision (s2) is available.

1.3.This document is meant for use within a single organisation, while the ISO-standard concerns interlaboratory experiments. This means that, either one laboratory will carry out the experiments, or else two laboratories of the same organisation, each laboratory being specialized in one of the methods.

1.4.As a consequence of point 1.3 here precision is not investigated under reproducibility conditions. Instead time-different intermediate precision conditions have been considered.

1.5.The ISO standard is meant to show that the two methods have similar precision and/or trueness. The present proposal is meant to show that the alternative method is at least as good as the standard method. This means that, in some instances where ISO applies two-sided tests, here one-sided tests are used.

1.6.Two different approaches are considered. The first one is based ,as in the ISO document, on the minimal number of measurements required to detect a specified bias between both methods and a specified ratio of the precision of both methods with high probability.

Since this might lead to a number of measurements to be performed that the laboratory considers too large, the second approach starts from a user-defined number of measurements. The probability () is then evaluated that an alternative method which is not acceptable, because it is too much biased and/or not precise enough, will be adopted.

This means that in both approaches an acceptable bias and an acceptable ratio of the precision measures of both methods have to be defined.

1.7.The evaluation of the bias is also based on interval hypothesis testing [1] in which the probability of accepting a method that is too much biased is controlled. The bias is considered acceptable if the one-sided 95% upper confidence limit around the estimated absolute bias does not exceed the acceptance limit for the bias.

2.Purpose of comparing measurement methods

The comparison of measurement methods will be required if a laboratory wishes to replace a method which is the recommended or official method in a particular field of application by an alternative method. The latter method should be at least as good (in terms of precision and trueness) as the first method.

3.Field of application

The document describes the comparison of the accuracy (trueness and precision) of two methods at a single concentration level. It is useful for comparisons at up to three concentration levels. Due to the problem with multiple comparisons [2] it should not be used if the methods are to be compared at more than three levels.

4.Accuracy experiment

4.1.General requirements

The procedures for both methods shall be documented in sufficient detail so as to avoid misinterpretation by the participating analysts. No modification to the procedure is permitted during the experiment.

4.2.Test samples

The precision of many measurement methods is affected by the matrix of the test sample as well as the level of the characteristic. For these methods, comparison of the precision is best done on identical test samples. Furthermore, comparison of the trueness of the methods can only be made when identical test samples are used. For this reason, communication between the working groups who conduct the accuracy experiments on each method should be achieved by appointment of a common executive officer.

The main requirement for a test sample is that it shall be homogeneous and stable, i.e., each laboratory shall use identical test samples. If within-unit inhomogeneity is suspected, clear instructions on the method of taking test portions shall be included in the document. The use of reference materials (RMs) for some of the test samples has some advantages. The homogeneity of the RM has been assured and the results of the method can be examined for bias relative to the certified value of the RM. The drawback is usually the high cost of the RM. In many cases, this can be overcome by redividing the RM units. For the procedure for using a RM as a test sample, see ISO Guide 33.

4.3.Number of test samples

The number of test samples used varies depending on the range of the characteristic levels of interest and on the dependency of the accuracy on the level. In many cases, the number of test samples is limited by the amount of work involved and the availability of a test sample at the desired level.

4.4.Number of measurements.

4.4.1.Determination of the minimal number of measurements required

In this approach the minimal number of measurements required, to detect a specified bias between two methods and a specified ratio of the precision of both methods with high probability, is determined.

4.4.1.1. General

The number of days and the number of measurements per day required for both methods depends on:

a) precisions of the two methods;

b) detectable ratio,  or , between the precision measures of the two methods; this is the minimum ratio of precision measures that the experimenter wishes to detect with high probability from the results of experiments using two methods; the precision may be expressed as the repeatability standard deviation, in which case the ratio is termed , or as the square root of the between-day mean squares, in which case the ratio is termed ;

c) detectable difference between the biases of the two methods, ; this is the minimum value of the difference between the expected values of the results obtained by the two methods that the experimenter wishes to detect with high probability from the results of an experiment.

It is recommended that a significance level of  = 0.05 is used to compare precision estimates and to evaluate the bias of the alternative method. The risk of failing to detect the chosen minimum ratio of standard deviations, or the minimum difference between the biases, is set at  = 0.20.

With those values of  and , the following equation can be used for the detectable difference:

(1)

where the subscripts A and B refer to method A and method B, respectively.

t/2 : two-sided tabulated t-value at significance level  and degrees of freedom

t : one-sided tabulated t-value at significance level  and degrees of freedom

: estimated variance component between days

: estimated repeatability variance component

p : number of days

n : number of measurements within one day

In most cases, the precision of method B is unknown. In this case, use the precision of method A as a substitute to give

(2)

The experimenter should try substituting values of nA, nB, pA and pB (and the corresponding t/2 and t) in equation (1) or (2) until values are found which are large enough to satisfy the value of  chosen (i.e. so that  computed with eq. 2 is smaller than the stated acceptable ).

It is strongly recommended to take nA = nB and pA = pB. In this case eq (2) simplifies to

(3)

NOTE : assuming sAto be equal to sBis of course a strong assumption since even ifA = B it is improbable for sA to be equal to sB. Therefore eqs.(2 and 3) are only approximates which could be further simplified by replacing (t/2 + t) by a constant value. Indeed for =0.05 and =0.20 , (t/2 + t) varies between 2.802 ( = ) and 3.195(=8 i.e. pA = pB = 5) and therefore a constant value equal to 3 could be used throughout. Equation (3) then becomes:

The values of the parameters which are needed for an adequate experiment to compare precision estimates should then be considered. Table 1 shows the minimum ratios of standard deviation for given values of  and  as a function of the degrees of freedom A andB.

For repeatability standard deviations

For between-day mean square :

If the precision of one of the methods is well established use degrees of freedom equal to 200 from Table 1.

Table 1

Values of  (A, B, , ) or  (A, B, , ) for  = 0.05 and  = 0.20

4.4.1.2. Example: Determination of iron in iron ores

NOTE : the example of ISO has been adapted to the situation where the precision is not known but estimated as s2and to the application of a one-sided test for the evaluation of the precision.

4.4.1.2.1. Background

Two analytical methods for the determination of the total iron in iron ore are investigated. An estimate of the precision (srA and stA) for method A is available. Both methods are presumed to have equal precision. Therefore

4.4.1.2 2. Requirements

The minimum number of days required are computed assuming equal number of days and duplicate analyses per day:

pA = pB and nA = nB = 2

a) For the trueness requirement :

With pA = 5, (t/2 + t) = 3.195 and =0.428; with pA = 6, (t/2 + t) = 3.107 and  = 0.381. Hence pA = pB = 6.

NOTE : the use of a constant multiplication factor equal to 3 would also yield pA = pB = 6p.

b) For the precision requirement :

From Table 1 it can be seen that =4 or =4 is reached when .

To compare repeatability standard deviations, .

To compare between-day mean squares,

.

4.4.1.2.3. Conclusions

The minimum number of days required (with two measurements per day) is 6.

This means that with this sample size (), provided that reliable precision estimates were considered

- the probability that it will be decided that there is a bias when in fact there is none is 5% and at the same time the probability that a true bias equal to 0.4% will go undetected is 20% and

- the probability that it will be decided that the precision measures of both methods are different when in fact they are equal is 5% and at the same time the probability that a true ratio between the precision measures of both methods equal to 4 will not be identified as being different is 20%.

4.4.2. User-defined number of measurements

This approach is based on a user-defined number of measurements. This means that the number of days () and the number of measurements per day () are defined by the user. It is however strongly recommended to take .

In the comparison of the results of method A and method B the probability  that an alternative method which is not acceptable, because it is too much biased and/or not precise enough, will be adopted is then evaluated. This probability depends on:

a) precisions of the two methods;

b) detectable ratio,  or , between the precision measures of the two methods; this is the minimum ratio of precision measures that the experimenter wishes to detect with high probability from the results of experiments using two methods; the precision may be expressed as the repeatability standard deviation, in which case the ratio is termed , or as the square root of the between-day mean squares, in which case the ratio is termed ;

c) detectable difference between the biases of the two methods, ; this is the minimum value of the difference between the expected values of the results obtained by the two methods that the experimenter wishes to detect with high probability from the results of an experiment;

d) the significance level  and the number of measurements.

4.5.Test sample distribution

The executive officer of the intralaboratory test programme shall take the final responsibility for obtaining, preparing and distributing the test samples. Precautions shall be taken to ensure that the samples are received by the participating analysts in good condition and are clearly identified. The participating analysts shall be instructed to analyse the samples on the same basis, for example, on dry basis; i.e. the sample is to be dried at 105°C for x h before weighing.

4.6.Participating analyst

The laboratory shall assign a staff member to be responsible for organizing the execution of the instructions of the coordinator. The staff member shall be a qualified analyst. If several analysts could use the method, unusually skilled staff (such as the "best" operator) should be avoided in order to prevent obtaining an unrealistically low estimate of the standard deviation of the method. The assigned staff member shall perform the required number of measurements under repeatability and time-differentconditions. The staff member is responsible for reporting the test results to the coordinator within the time specified.

It is the responsibility of this staff member to scrutinize the test results for physical aberrants. These are test results that due to explainable physical causes do not belong to the same distribution as the other test results.

4.7.Tabulation of the results and notation used

With 2 measurements per day (n=2), as recommended, the test results for each method can be summarized as in Table 2 where:

p / is the number of days
/ are the two test results obtained on day
/ is the mean of the test results obtained on day

/ is the grand mean

Table 2

Summary of test results (e.g. for Method A)

DayTest resultsMean

1

4.8.Evaluation of test results

The test results shall be evaluated as much as possible using the procedure described in ISO 5725-2 and ISO 5725-3.

This includes among others that outlier tests are applied to the day means.

4.8.1. Outlying day means

4.8.1.1. One outlier

The day means are arranged in ascending order.

The single-Grubbs' test is used to determine whether the largest day mean is an outlier. Therefore the Grubbs' statistic G is computed:

where

To determine whether the smallest day mean is an outlier compute Grubbs' statistic G as follows:

Critical values for Grubbs' test are given in the Appendix I.

If at the 5% significance level G  Gcrit, no outlier is detected.

If at the 1% significance level G > Gcrit, an outlier has been detected. It is indicated by a double asterisk and is not included in further calculations.

If at the 5% significance level G > Gcrit and at the 1% significance level

G  Gcrit, a straggler has been detected. It is indicated by a single asterisk and is included in the further calculations unless the outlying behaviour can be explained.

4.8.1.2. Two outliers

If the single-Grubbs' test does not detect an outlier, the double-Grubbs' test is used to determine whether the two largest day means are outliers. Therefore the Grubbs' statistic G is computed as follows:

where

and

To determine whether the two smallest day means are outliers the Grubbs' statistic G is computed as follows:

where

and

Critical values for the double-Grubbs' test are also given in the Appendix I. Notice that here outliers or stragglers are detected if the test statistic G is smaller than the critical value. The outliers found are indicated by double asterisk and are not included in further calculations. The stragglers found are indicated by a single asterisk and are included in the further calculations unless the outlying behaviour can be explained.

4.8.2. Calculation of variances

A summary for the calculation of the variances is given in Table 3.

For each test sample, the following quantities are to be computed:

/ is an estimate of the repeatability standard deviation for method A
/ is an estimate of the repeatability standard deviation for method B
/ is an estimate of the time-different intermediate precision standard deviation for method A
/ is an estimate of the time-different intermediate precision standard deviation for method B ()

Table 3

Calculation of variances

ANOVA table

Source / Mean squares / Estimate of
Day / /
Residual / /
Calculation of variances
- The repeatability variance
/ df = p (n-1)
- Variance component between days (between-day variance)
- Time-different intermediate precision (variance)
- Variance of the means
/ df = (p-1)

4.9.Comparison between results of method A and method B
The results of the test programmes shall be compared for each level. It is possible that method B is more precise and/or biased at lower levels of the characteristic but less precise and/or biased at higher levels of the characteristic values or vice versa.

4.9.1. Graphical presentation

Graphical presentation of the raw data for each level is desirable. Sometimes the difference between the results of the two methods in terms of precision and/or bias is so obvious that further statistical evaluation is unnecessary.

Graphical presentation of the precision and grand means of all levels is also desirable.

4.9.2. Comparison of precision

NOTE : since we want to evaluate whether the alternative method is as least as good as the reference method the hypotheses to be tested are .

4.9.2.1. Based on the minimum number of measurements required

4.9.2.1.1. Repeatability

If

(4)

there is no evidence that method B has worse repeatability than method A;

if

there is evidence that method B has worse repeatability than method A.

is the value of the F-distribution with rB degrees of freedom associated with the numerator and rA degrees of freedom associated with the denominator;  represents the portion of the F-distribution to the right of the given F-value and

rA = pA(nA - 1)

rB = pB(nB - 1)

4.9.2.1.2. Time-different intermediate precision

For the comparison of the time-different precision the number of degrees of freedom associated with the precision estimates is needed. Since these estimates are not directly estimated from the data but are calculated as a linear combination of two mean squares, MSD and MSE (see Table 3), the number of degrees of freedom are determined from the Satterthwaite approximation [3].

However to avoid the complexity in the determination of the degrees of freedom associated with , the comparison of the time-different intermediate precision can be performed in an indirect way by comparing the variance of the day means , provided that the repeatabilities of both methods are equal and the number of replicates per day for both methods is equal .