Peer Review of Defra Funded Science

Peer Review of Defra Funded Science

Final Report Appraisal

Project Title : New Approaches to Evaluating and Quantifying the Benefits of Chemicals Regulation

This form is to be used for the peer review of completed final reports of Defra-funded science. When you have completed this form (preferably typed), please sign it and send it via post to Defra at the address given at the end of the form. If you have any questions relating to the completion of this form please contact either:

Victoria Cox - Tel 0207 082 8594

Kate Perry – Tel 0207 082 8085

Instructions

If, for whatever reason, you feel that you are not able to confidently assess the report(s) you have been sent, please advise Defra as soon as possible.

Consider each question on the form and tick the box that most represents the answer to the question for the report being reviewed.

Given the importance of high quality refereeing, it is essential that you can speak with confidence with respect to the project report(s) you have received, and that you can justify your comments. It may be that you feel you are only able to comment on some aspects of the report. Therefore only answer questions on which you feel you are able to give a qualified judgement or are applicable.

A full justification for each answer must be included in the ‘Comments’ box. The space allocated in the box is for guidance only. If you are using this form electronically the boxes are expandable, alternatively if you are using a hard copy, feel free to use continuation sheets. When using continuation sheets, please number and mark them clearly with the question number.

This form and any continuation sheets may be passed to the researcher responsible for the project and to the funding policy division within Defra. Would you prefer your name NOT to be attributed to this form?

REPORTING AND METHODS

1. Scope and Objectives. Does the report address all aspects of the objectives of the study stated in the agreed specification?

YES □ NO X

If YES, please go to question 2.

If NO, are: Tick ONE box only

(a) all objectives stated and most addressed satisfactorily?...... □

(b) objectives broadly stated but not all clearly addressed?...... X

Please comment:

The study would appear to have addressed objectives 1 and 2 as set out in the terms of reference.
With regard to Task 1 of Objective 3, in my view this has not been fully met. Because of the case studies that were selected and the fact that the approach relies on the use of hazard data from EUSES/ESR RARs, it has not dealt with some of the direct costs that may arise from chemicals in the environment. These may be particularly important in relation to future action aimed at limiting vPvBs, for example, where there is no clear toxicity endpoint. For this set of chemicals, the approach proposed here has no applicability.
In addition, it is not clear that the range of required EUSES input data would be available at a screening/prioritisation stage. The PBT screening that is currently undertaken by the ACHS may be based on minimal toxicological data, for example, acute toxicity for algae. Furthermore, there may also be incomplete data on levels of usage across different sectors at a screening/ prioritisation stage. Considerable work may also be required to develop estimates of the reductions in emissions that would result from anything other than restrictions on production, marketing and use. This obviously does not invalidate the approach but risks the danger that it is used as an IA tool rather than a prioritisation tool. Use for IAs may be appropriate where it should be possible to modify some of the simplifying assumptions and there is some means of validation; however, it also implies that further work would have to be undertaken to develop more robust and credible chemicals specific valuation data to provide the basis for economic analysis.
It would appear to have generally fulfilled Task 2 under Objective 3, but to a degree lacks some clarity with regard to some of the assumptions. Further work could also be put into discussing the wider literature, in particular in relation to health valuation. For example, it is not clear why some of the UK health valuation literature and the values used for recent UK air quality policy have been rejected in preference for values developed by the Danish Environment Ministry (see below).

2. Quality of Approach (i.e. development of the model). Do the approach and methodology adequately address the objectives? Are there any weaknesses that could cast doubt on the conclusions?

(a) Quality of approach is sound and robust. It is optimal for the research conducted throughout the project……………………………………………………………………………………………□

(b) Quality of approach is generally sound. Some parts weaker than others ……………...X

(d) Approach is such that the conclusions could be flawed…………………………………….□

Please comment:

Note that I also answer questions 1, 3 and 4 of the additional issues here.
If the intention is to use the methodology for prioritisation only, and it is not to also provide the basis for the estimation of benefits as part of a risk reduction strategy then it could be considered to be generally sound.
In some ways the approach is very attractive as it provides a means of ranking pollutants according to their economic ‘damage potential’. It is limited though to application of chemicals that have either an environmental toxicity endpoint or a human health toxicity endpoint. It is not applicable to vPvBs and may not be applicable to some of the other potential substances of very high concern (e.g. it does not include valuations for mutagens or reprotoxins at present).
More importantly though if the methodology is to be used as part of IAs and Restrictions Dossiers under REACH, it is likely to be challenged due to the fact that it does not rely on dose-response relationships. In other words it is not risk based. Industry’s mantra that regulation must be risk based and not hazard based will be levied against this approach. This is in part due to the reliance on assessment safety factors within the calculations of PNECs and NoELs. 100% of the population may be protected at concentration values close to the LC50/EC50 value, rather than at levels three orders of magnitude lower. Industry is therefore likely to challenge the validity of a safety factor based number of valuation purposes and argue the importance of dose-response relationships.
Also because of the many simplifying assumptions, the approach may provide an unrealistic assessment of the benefits of restricting a particular chemical. The simplifications necessarily inherent to the model may mean that some of the outputs are not reliable. Building on the above argument, the LC50/EC50 data that underlie calculation of the PNECs are for the most sensitive species within the ‘continental EU’ ecosystem although, in reality, those species may not even exist in the correlating EU ecosystem (by way of example, the most sensitive snail used in some of the tests to determine endocrine disruption is a warm water snail that is not found in EU waters). Thus, linking damage to the most sensitive species and then translating this to other species in the ecosystem may not be appropriate. The question is whether it is really scientifically robust to assume that the chronic toxicity threshold for the most sensitive test species is the chronic toxicity threshold for an ecosystem.
In relation to the WFD it has been argued that based on a comparison of PNECs as calculated following the TGD and given the levels of some of the metals and other substances currently found in the aquatic environment, it is surprising that we have any high quality fisheries (salmonid and coarse) in UK rivers. This type of conclusion is even repeated in this study, with the note that ‘run of river’ commercial trout farms are not affected by chemicals while it is assumed that wild trout living in the same river would be.
I did not really understand the need for the impact scores in addition to the impact descriptions and effects which are in any event described in terms of percentage of population affected. It seems like an unnecessary interim step that leads to some confusion when looking at tables etc in the annexes. I also found myself wanting to combine the impact levels with the effects categories for a more sophisticated indicator or impact – this was particularly true for health, where it may be more valid to consider 5% of the population at risk from a low risk health impact such as sensitisation. In other words, I would want to treat the impact levels and the effects categories as two independent variables that would then be combined. This may help enable the approach to better allow for the fact that many of the health concerns are in relation to sensitive populations, such as the elderly or children.
I had no problems with the approach for characterisation of the local environment, nor with the approach to monetary valuation in general although I would agree with the authors that the existing set of valuations is probably not a very robust set for chemicals policy.

3. Assumptions. Are any assumptions made during the work sound and clearly identifiable?

(a) Assumptions are clearly identified and sound………………………………………………□

(b) Assumptions are identifiable and are broadly in line with current thinking and/or are justifiable in the particular circumstances…………………………………………………………………….X

(d) Assumptions are not identified and/or are based on unsound judgement……………….□

Please comment:

Note that I also answer questions 2 and 5 here.
A series of different assumptions are made in the approach. Some of these are in my view more robust than others.
1) Use of PNECs
In Section 1.5 the meaning of a LC50 is stated as the lethal concentration at which 50% of the population is affected. However, these values are then assumed in the scoring, etc. to relate to 100% of the population dying.
Note that not all endpoints involve the use of a safety factor of 1000 – some adopt a factor of only 100. Further development of the methodology would need to address this aspect.
2) Use of RCRs for assessing impacts. RCRs are hazard quotients and are recognised as such. Much of the literature on chemical risk management notes that there is no simple step in moving from these to dose-response functions and estimates of the population actually at risk (or at different levels of risk) so as to enable valuation of the incremental benefits of different policies. Thus, basing the analysis on these does miss this fundamental linkage. In some cases, it may be possible to make these links, for example, in relation to trichloroethylene and its role in VOC formation and hence in ozone. The dose-response data in this case exist for at least partial valuation, e.g. as in for the UK Air Quality Strategy.
4) Theoretical ‘dose-response relationship’. The approach essentially creates a surrogate dose-response function based on the use of the assessment factors (and indeed the tool refers to it as a ‘dose response function’). It is not clear to me what the advantages are of the five step ‘function’ over the use of an equation in the case of environmental valuation. The authors in any case end up interpolating values between their different impact scores. This would also address some of the issues raised in the report with regard to whether the bands of 5%, 10% and 20% are the most appropriate; it would remove the arbitrariness of this aspect.
3) Assessment period of 15 years. In my view an assessment period of 15 years may not be appropriate to the types of chemicals being considered here, given that they are likely to be PBTs, vPvBs and CMRs. Obviously, one does not want to adopt a ridiculously long time period, and some judgement has to be made here as to what is appropriate.
Linked to this though is the question of how the benefits of the policy are aggregated. For some of the health effects, the benefits of regulation may occur immediately and be fully realised within a short period of time (e.g. within one or two years). In other cases, the benefits may all be realised in a short period of time.
Something that can’t be seen from the tool as it was provided in read only format is how the method aggregates cancers versus acute effects. The number of acute effects avoided per annum is presumably a constant, while the number of VOLYs gained from reduction of cancers would be an increasing number every year?

DATA AND ANALYSIS

4. Evidence Base. Does the evidence on which the analysis is based draw on appropriate, recent and relevant studies in this field? Is the evidence considered representative of the evidence that exists? Is any evidence ignored or under-represented?

(a) Evidence is drawn from appropriate, recent and relevant studies………………………..□

(b) Evidence is mostly drawn from appropriate, recent and relevant studies………………..X

(d) Evidence is not representative of the evidence that exists………………………………...□

Please comment:

Note that I also answer question 6 here
1) Hazard Data. The study draws on the most appropriate hazard data across the range of relevant endpoints. However, some validation could have been carried out for trike (trichloroethylene) in relation to the various studies that have been undertaken to assess the impacts of the VOC directive.
2) Environmental Valuations. With regard to environmental valuation, I would agree that the authors have used the most appropriate set of valuations. There are other marine fishery studies, but I do not think that they would improve the robustness of the model. The general point is that there are no really reliable values for benefits transfer of this kind in relation to the environment. The inland recreational fisheries and non-use values are probably OK but I really doubt the validity of the informal recreation and the marine values – even if they are the best available.
I have not re-looked at the marine fisheries study by Drew Associates, but I am surprised that the participation rate for this group of recreational anglers is so high that it translates to 6.8 days per annum per household (mind you I got confused as to how this study was used and the figures given in Table B4 don’t seem to match the text in Section 2.2.3).
I admit to also being surprised by the valuation of the trike benefits in relation to marine fisheries. These benefits come out some much higher than the crop benefits which would normally be linked to a VOC such as this and to the inland fisheries benefits. Also, I note that man via the environment is not considered here – which is surprising as it is a carcinogen, VOC, etc.
3) Health Valuations. I found the review of the health literature weak – and the lack of a reference list/bibliography in the report made it difficult to know what studies were considered. I am surprised that there is no reference to the work carried out by the UK’s Health Economics Advisory Panel with regard to valuation of health effects. It is not clear why the Danish study is adopted in preference to equally recent work undertaken by Chilton et al 2004, the work done by AEA in 2006 for Defra on the air quality strategy, or to the various studies which have placed economic valuations on the health benefits stemming from the VOC Directive.
I have not examined the Danish study in detail, but there are issues in transferring health care costs (i.e. the resource cost element) from one country to another due to differences in what is covered, the element of these costs that is a transfer versus a real resource cost, etc. This type of issue is discussed by the ENVECO paper on the valuation of a statistical life (the website is under ‘maintenance’ so I cannot find a reference. This paper looks at the differences in resource costs across EC member states and provides discussion on valuations of VOLYs for cancer, etc. Similarly, there is no reference to the work by Pickvance et al (2005) for the ETUC on the occupational health benefits of REACH which provides estimates for endpoints such as sensitisation.

5. Uncertainties. Are uncertainties in the data identified and recognisable?

(a) Uncertainties in the data are clearly identified and explained. ……………………………X

(b) Uncertainties in the data are broadly identified although greater clarification

would be beneficial……………………………………………………………………………...

(d) It is not possible to identify where there may be uncertainties in the data ...... ……..□

Please comment:

Uncertainties are flagged together with potential areas where the method could be improved. Some of the discussion could be expanded but this is not a significant fault of the study outputs. The authors are aware of the weaknesses of the approach and identify these in numerous places.

6. Analysis. Is the analysis sound, clear and appropriate for the work undertaken?