ADDITIONAL FILE 1: ELEMENTS OF A PROTOCOL FOR DOCUMENTARY ANALYSIS

We set out below the possible elements of a protocol for the documentary analysis that would be undertaken in each study to attempt to identify the degree of consistency between the policy and the body of relevant international research that is being centrally collated by the WHO utilisation project team. (Additional, more routine, documentary analysis would be undertaken as appropriate, for example, as part of the preparation for interviews and when collating data for the application of the scales). Before being implemented, the protocol would need to be reviewed and agreed between the assessment teams from each country. Some of the key steps would be:

  1. Clarification of who would undertake the identifications and comparisons described below: either the assessment team or, if necessary, experts in the particular field. At least two opinions would be sought for each policy document.
  1. Identification of the appropriate policy documents. This would vary by level of policy-making (national/sub-national, institutional, clinical unit or professional body) and also by type of policy. The spectrum of policy types would be reflected in the spectrum of possible policy documents, which includes: legislation, administrative/executive regulations or orders, reimbursement arrangements, guidelines/advice etc.
  1. Examination of the consistency of content between the policy documents and the international research findings, taking account of the following issues:

The amount of the policy that was consistent with the research;

How far the policy included the key issues covered by the research (as identified by the central analysis of the body of research);

How far the policy included the elements of research regarded as providing the strongest evidence (as identified by the central analysis using, where appropriate, grading criteria);

How far the policy was consistent with research in terms of: the definition of the policy problem; definition of objectives; and the description of strategies and actions;

How far elements of the policy contradicted the research evidence.

  1. Initial application of the scale of the degree of consistency between policy and research—see Draft Scales of the Level of Research Utilisation in Health Policy-Making. In applying the scale, the five factors listed in point 3 above would be taken into account.Where the policy consists of a clinical guideline developed by a professional group there could be circumstances in which this scale on the degree of consistency, based on documentary analysis, might be the only scale appropriate to apply. In such circumstances, the analysis should probably concentrate on the quality of the evidence used in the guidelines. Here again grading criteria could become important. If the main assessment for clinical guidelines is going to be through documentary analysis, it would probably be worth extending the focus beyond the international body of research literature that it is proposed be used for most documentary analysis. In addition, local research should probably be included because it could be important to see how far attempts had been made to ensure the guidelines were relevant for local circumstances.