Home | Advanced & = Citation=20 Search | Help

The
Journals

Exclusively
Online

Article
Collections

Authors

Jobs=20 &
Conferences

About = Us

My = Account

Mr. de straling

The Lancet, Volume = 369, Number=20 9576, 2=20 June 2007

Contents=20 in Full

Editorial
Comment
World=20 Report
Perspectives
Obituary
Correspondence
Department=20 of Error

Articles
Clinical=20 Picture

Seminar
Case=20 Report

The=20 Lancet 2007; 369:1883-1889

DOI:10.1016/S0140-6736(07)60675-8

Articles

Use of evidence in WHO=20 recommendations

Dr=20 Andrew D Oxman MD a , John=20 N Lavis MD b and Atle Fretheim=20 MD a

See Comment

Summary

Background

WHO=20 regulations, dating back to 1951, emphasise the role = of expert=20 opinion in the development of recommendations. = However, the=20 organisation's guidelines, approved in 2003, emphasise = the use=20 of systematic reviews for evidence of effects, = processes that=20 allow for the explicit incorporation of other types of = information (including values), and evidence-informed=20 dissemination and implementation strategies. We = examined the=20 use of evidence, particularly evidence of effects, in=20 recommendations developed by WHO = departments.

Methods

We=20 interviewed department directors (or their delegates) = at WHO=20 headquarters in Geneva, Switzerland, and reviewed a = sample of=20 the recommendation-containing reports that were = discussed in=20 the interviews (as well as related background = documentation).=20 Two individuals independently analysed the interviews = and=20 reviewed key features of the reports and background=20 documentation.

Findings

Systematic = reviews and concise summaries of findings are rarely = used for=20 developing recommendations. Instead, processes usually = rely=20 heavily on experts in a particular specialty, rather = than=20 representatives of those who will have to live with = the=20 recommendations or on experts in particular = methodological=20 areas.

Interpretation

Progress=20 in the development, adaptation, dissemination, and=20 implementation of recommendations for member states = will need=20 leadership, the resources necessary for WHO to = undertake these=20 processes in a transparent and defensible way, and = close=20 attention to the current and emerging research = literature=20 related to these processes.

Back=20 to top

<= /DIV>

Introduction

Every=20 year, WHO develops a large number of recommendations = aimed at=20 many different target audiences, including the general = public,=20 healthcare professionals, managers working in health=20 facilities (eg, hospitals) or regions (eg, districts), = and=20 public policymakers in member states. These = recommendations=20 address a wide range of clinical, public health, and = health=20 policy topics related to achieving health goals. WHO's = regulations emphasise the role of expert opinion in = the=20 development of recommendations. In the 56 years since = these=20 regulations were initially developed, research has = highlighted=20 the limitations of expert opinion, which can differ = both=20 across subgroups and from the opinions of those who = will have=20 to live with the consequences.1=E2=80=938=20 Experts have also been known to use non-systematic = methods=20 when they review research, which frequently result in=20 recommendations that do not reflect systematic = summaries of=20 the best available evidence.9,10

Evidence=20 of the effects of alternative policies, programmes, = and=20 services is essential for well-informed decisions. = Systematic=20 reviews have several advantages over other approaches = to=20 amassing evidence of effects.11=E2=80=9313 Firstly, = systematic reviews reduce=20 the risk of bias in selecting studies and interpreting = their=20 results. Secondly, they reduce the risk of being = misled by the=20 play of chance in identifying studies for inclusion, = or the=20 risk of focusing on a limited subset of relevant = evidence.=20 Thirdly, systematic reviews provide a critical = appraisal of=20 the available evidence and place individual studies or = subgroups of studies in the context of all the = relevant=20 evidence. Finally, they allow others to critically = appraise=20 the judgments made in study selection and the = collection,=20 analysis, and interpretation of the results. However,=20 systematic reviews are only as good as the evidence = that they=20 summarise. There might be no evidence. When there is = evidence,=20 judgments are still needed about the quality and, = especially=20 for public health and health policy topics, its = applicability=20 in different contexts.12

Evidence=20 of effects needs to be complemented by information = about=20 needs, factors that could affect whether effectiveness = will be=20 realised in the field, such as the available = resources, costs,=20 and the values of those who will be affected by the=20 recommendations. Processes that allow for the explicit = incorporation of these types of information, = particularly=20 values, have (like systematic reviews) emerged as = central to=20 the development of recommendations.14=E2=80=9318 Moving from = evidence to=20 recommendations requires judgments, particularly = judgments=20 about goals and about the balance between the = desirable and=20 undesirable consequences of choosing one option over = another=20 to achieve these goals.

Evidence-in= formed=20 dissemination and implementation strategies are = increasingly=20 recognised as a core part of the business of = developing=20 recommendations. Those charged with developing = clinical=20 practice guidelines can draw on a systematic review of = randomised controlled trials of guideline = dissemination and=20 implementation strategies to inform their efforts.19,20 Although there are no easy = solutions=20 and few strategies have been assessed in low-income = and=20 middle-income countries, such efforts clearly can have = an=20 effect.21=20 Those charged with developing recommendations targeted = at=20 managers or public policymakers, on the other hand, = have to=20 deduce the attributes of the interventions from = systematic=20 reviews of observational studies and begin to build an = evidence base about the effectiveness of these=20 interventions.13,22

WHO=20 has recognised the need to revise its approach to = developing=20 recommendations, in guidelines approved by the WHO = Cabinet in=20 2003.23=20 We sought to examine the use of evidence in WHO=20 recommendations subsequent to this. We particularly = wanted to=20 explore the use of evidence of effects. Our hope was = that such=20 stock-taking would inform debates about how WHO could = improve=20 how it develops and disseminates recommendations and = how WHO=20 could better support member states in their efforts to = adapt=20 and implement recommendations.

Back=20 to top

Methods

We=20 interviewed department directors (or their delegates) = at WHO=20 headquarters and reviewed a sample of the=20 recommendation-containing reports that were discussed = in the=20 interviews. We invited the participation of all = department=20 directors in five departmental clusters that had a = content=20 focus: non-communicable diseases and mental health = (six=20 departments); HIV/AIDS, TB, and malaria (four); family = and=20 community health (four); communicable diseases = (three); and=20 health technology and pharmaceuticals (two). We = invited the=20 participation of one department (of five) in the = sustainable=20 development and healthy environments cluster and three = departments (of five) in the evidence and information = for=20 policy cluster. We did not invite the participation of = the=20 department directors in the two clusters=E2=80=94the = external=20 relations and governing bodies cluster and the general = management cluster=E2=80=94that had a corporate focus. = Although our=20 written request for an interview was introduced by a = WHO=20 department director, we made clear that the study was=20 independent of WHO and that we planned to publish the = results=20 after first making them available to WHO. We = purposively=20 sampled four of the reports identified by those = interviewed on=20 the basis of their focus on clinical treatment, = centrality to=20 major WHO initiatives, and relevance to the Millennium = Development Goals. The first criterion was chosen to = maximise=20 the chances that evidence of effects would be = available and=20 that the reports could be expected to meet current = standards=20 for clinical practice guidelines.

Two=20 individuals participated in each interview. One = individual had=20 main responsibility for doing the interview and the = other for=20 recording the interview on audiotape and taking notes. = The=20 brief structured part of each interview focused on the = number=20 and background of staff members and the number and = type of=20 recommendation-containing reports published in the = past year.=20 The semi-structured part of each interview focused = mainly on=20 the development of recommendations contained in one or = two=20 specific published guidelines or policies that were = selected=20 by interviewees from among those their department had=20 developed or had a major responsibility in developing. = For=20 each guideline or policy, we asked about: why it was=20 developed; the process used (including whether support = was=20 received from others within or outside WHO, whether = evidence=20 of effects and other types of information were used, = whether=20 and how supporting documentation was made publicly = available,=20 and whether and how plans for updating were = established);=20 strengths of the processes used and elements that = could have=20 been improved; likely benefits, harms and costs of = adhering to=20 the recommendations; how the recommendations have been = used=20 and any plans for assessing the effects of adherence = to the=20 recommendations; and the availability of any = background=20 documentation.

Two=20 individuals independently did a thematic analysis of = the=20 interviews and reviewed key features of the=20 recommendation-containing reports (and related = background=20 documentation). We began the thematic analysis by = using the=20 notes taken during each interview (supplemented by the = corresponding audiotape) to produce a summary of each=20 interview, including the major themes that emerged. We = then=20 sent the summary to each interviewee with a request = that they=20 verify our interpretations and, if they wished, = provide=20 additional comments. We used the audiotapes to = identify=20 illustrative quotations for each major theme. We began = the=20 review by recording for each document its type, = whether it=20 included a section that described the methods used, = the number=20 of recommendations that were based on a systematic = review, the=20 number of systematic reviews cited, and the = description=20 provided of the development process. We then produced = a=20 summary for each recommendation-containing report. We=20 presented our findings at various forums within WHO as = an=20 additional check on our interpretations.

The=20 study was sponsored as part of a broader = project=E2=80=94Pragmatic=20 Randomised Controlled Trials In Health = Care=E2=80=94funded by the=20 European Commission's 5th Framework International=20 Collaboration with Developing Countries. WHO was a = formal=20 partner in Practihc; however, WHO staff input was = limited to=20 commenting on the protocol and interview questions, = and=20 providing comments on the interview summaries (for = those who=20 were interviewed) and overall findings (for those who = attended=20 forums where these were presented).

We=20 did 23 interviews with 29 people, and reviewed four=20 recommendation containing reports and related = background=20 documentation. We interviewed the director in 15 = departments=20 and someone designated by the director in six = departments.=20 Three of the interviews were with more than one = person. For=20 two departments we did two separate interviews with = different=20 people from the department. We were unable to arrange=20 interviews with the directors (or delegates) of two=20 departments that had newly appointed directors who = were not=20 yet in post. The interviews, which lasted for up to an = hour,=20 were done between September, 2003, and February, 2004. = Five of=20 21 participating departments did not produce formal=20 recommendations and so their interviews were not = included in=20 the analysis, which is therefore based on 17 = interviews with=20 21 people (across 16 departments). The four=20 recommendation-containing reports that we selected for = review=20 were clinical practice guidelines that addressed=20 antiretroviral therapy for HIV, treatment of = tuberculosis,=20 treatment of malaria, and the integrated management of = childhood illness.24=E2=80=9327

Role of the = funding=20 source

The=20 sponsors of the study had no role in the design = (beyond=20 commenting on the protocol and interview questions), = data=20 collection, data analysis, data interpretation (beyond = commenting on the interview summaries and overall = findings),=20 writing or revising of the report. The corresponding = author=20 had full access to all data in the study and had final = responsibility to submit the report for = publication.

Back=20 to top

<= /DIV>

Results

The=20 directors or their delegates (hereafter directors) of = the 16=20 departments that developed recommendations reported = that their=20 departments had between eight and 170 staff members = each=20 (median 55) and close to 1000 staff members in total. = The=20 directors estimated that between 20% and 80% of staff = members=20 had some background in research (median 30%). Many = directors=20 had difficulty quantifying the number of=20 recommendation-containing reports that their = department=20 published each year because of the various formats of = the=20 recommendations. Their estimates ranged from one to 45 = reports=20 per department per year (median 8), with a total of = almost 180=20 reports per year.

The=20 reports varied widely in the nature of the topics they = addressed. In addition to clinical treatment topics, = the=20 reports addressed topics such as malaria control with=20 insecticide-treated bednets, promotion of mental = health,=20 helminthic guidelines for managers, human resources = policy=20 development, model list of essential medicines, = tobacco=20 legislation, and bioterrorism. The directors cited = several=20 reasons for developing recommendations, the most = common of=20 which were a perceived need for guidance, a perceived = need for=20 updating existing recommendations, and demand from = member=20 states. One report was developed to respond to = criticisms of=20 previous recommendations.

Expert=20 committees or meetings of experts were almost always = convened=20 when developing recommendations whereas only a few = directors=20 mentioned having commissioned systematic reviews to = inform the=20 work of these expert groups. Some directors reported = the use=20 of a combination of work done in-house and an expert = committee=20 or the combination of a small task force to draft=20 recommendations and either an expert committee or a = review by=20 external experts. Many directors reported a phase of = external=20 consultation or review. Only a few directors mentioned = developing dissemination or implementation strategies. = Most=20 directors reported the involvement of one or more = other WHO=20 departments in the development process, and nearly all = reported some form of external support. No directors = mentioned=20 drawing on any form of internal support in the = methodological=20 or technical aspects of developing recommendations. = The=20 external support typically took the form of expert = committee=20 members, but sometimes involved expert advisors, = writers of=20 background reports and recommendation-containing = reports, and=20 reviewers.

When=20 asked specifically about using evidence of effects, = only a=20 small number of directors reported using systematic = reviews of=20 such evidence and none reported using concise = summaries of=20 findings (eg, balance sheets) for the most important = outcomes=20 (benefits, harms, and costs) of each option being = considered.=20 Many directors instead reported using background = documents,=20 although there was little consistency in how the = documents=20 were prepared. For example, some background documents = were=20 prepared by the participating experts according to = their own=20 conventions. Other directors reported leaving the use = of=20 evidence up to the experts, feeling that evidence of = effects=20 was not relevant for some recommendations, and feeling = that=20 randomised trials were not appropriate for some types = of=20 interventions. Only one director reported grading the = quality=20 of the evidence.

When=20 asked about the use of other types of information, = several=20 directors reporting using data about costs but only a = couple=20 mentioned using data about potential harms or = explicitly=20 considering values=E2=80=94ie, the relative importance = or worth of the=20 consequences (benefits, harms, and costs) of a = decision.18=20 Using data about potential harms was only mentioned in = relation to clinical interventions, particularly=20 pharmaceuticals, and not for public health or policy=20 interventions. Explicitly considering values was = undertaken in=20 a general way. One director talked about the = =E2=80=9Cweighing of=20 values, which basically reflected the composition of = the=20 panel.=E2=80=9D Another director commented: = =E2=80=9CValues were also brought=20 into debate. For example, experience for high income = countries=20 suggest that encouraging more self efficacy and = independency=20 for young people could be effective in preventing = mental=20 health problems and substance use. However, this was = by many=20 considered as to be contrary to important values for = people=20 living in many low-income countries.=E2=80=9D

Although=20 directors were not asked specifically about group = processes,=20 many volunteered descriptions that suggested that = these=20 processes were not particularly structured with = respect to=20 group composition, format, or rules. These = descriptions=20 suggested that participants were implicitly weighing = evidence=20 of effects, harms, and costs along with values and = many other=20 types of information (eg, surveys, resistance = patterns, other=20 epidemiological data, availability of interventions, = country=20 experiences, political considerations, cultural = differences,=20 ethical considerations, and undocumented knowledge). = One=20 director clearly recognised the challenges associated = with a=20 lack of structured process: =E2=80=9CThere is a = tendency to get people=20 around the table and get consensus=E2=80=94everything = they do has a=20 scientific part and a political part. This usually = means you=20 go to the lowest common denominator or the views of a = =E2=80=9Cstrong=E2=80=9D=20 person at the table.=E2=80=9D

Most=20 directors reported that the information that was used = by the=20 committees was not published but was often made = publicly=20 available in some form. The format for the = documentation=20 varied widely, including a bibliography in the report, = one or=20 more published articles (such as a special edition of = the Bulletin of the World Health = Organization), one=20 or more reports (eg, annual reports, multicountry = assessment=20 reports, and proceedings of meetings), a book, and an=20 adaptation guide. The documentation was sometimes = readily=20 available (eg, on a website) and other times required = personal=20 contact with those involved in developing the = recommendations.=20 Although one director reported updating = recommendations every=20 2 years and some other directors reported that their=20 recommendations were considered one-off initiatives = that would=20 not be updated, most directors reported ad hoc = approaches to=20 deciding whether and when to update recommendations. = One=20 director reported plans to update the recommendations = using=20 the guidelines for WHO guidelines.

The=20 directors identified numerous strengths in the = processes used=20 for developing recommendations and in the = recommendations=20 themselves. The most commonly identified strength was = bringing=20 together or consulting with a wide range of people. = Most other=20 strengths were mentioned by only one director, = although=20 several of these strengths can be grouped together (panel 1).

Panel 1: Strengths in WHO=20 processes as identified by directors

=E2=80=A2Usefulness of = the=20 recommendations, which included attributes like = focusing on=20 end users, ensuring usability, responding to the = concerns of=20 donors, and filling a gap

=E2=80=A2Evidence-based = process,=20 which included attributes like obtaining evidence in a = rigorous way, drawing on good data, basing = recommendations on=20 research, using cost-effectiveness analyses, testing = the=20 recommendations, and doing validation=20 studies

=E2=80=A2Experience-based process,=20 which included attributes such as involving people = with=20 practical experience and, although this was also = considered a=20 weakness, developing instinct-based=20 recommendations

=E2=80=A2Expert-based = process,=20 which included attributes like working with = knowledgeable=20 experts and obtaining consensus among=20 experts

=E2=80=A2Systematic = approach,=20 which included attributes such as using a standardised = method=20 and adopting so-called =E2=80=9Cguideline = logic=E2=80=9D rather than=20 =E2=80=9Ctechnocratic=E2=80=9D = approach

=E2=80=A2Group members = without=20 conflicts of interest

=E2=80=A2Good group = process as a=20 key element of the meeting = structure

=E2=80=A2Up-to-date=20 recommendations

Although=20 most directors identified one or more ways in which = the=20 recommendation-development process could have been = improved,=20 four did not identify any way in which improvements = could have=20 been made. Directors singled out the use of evidence = more=20 commonly than any other area for improvement (panel 2).

Panel 2: Comments by=20 directors

Comments=20 included:

=E2=80=A2=E2=80=9CI = would have liked to=20 have had more evidence to base recommendations on. We = should=20 have conducted a literature = search.=E2=80=9D

=E2=80=A2=E2=80=9CWe = never had the=20 evidence base well documented. We should have reviewed = evidence at a very early = stage.=E2=80=9D

=E2=80=A2=E2=80=9CThe = lack of resources=20 does limit the ability to develop evidence-based=20 recommendations.=E2=80=9D

=E2=80=A2=E2=80=9C[Director General]=20 Brundtland came in and said =E2=80=9Cevidence, = evidence, evidence=E2=80=9D but=20 the approach to expert committees hasn't changed since = the=20 1950s=E2=80=94many see WHO as a technical agency and = therefore we=20 should have a comprehensive review of recommendation=20 processes, including expert=20 committees.=E2=80=9D

=E2=80=A2=E2=80=9CMaybe = what WHO needs is=20 more work on the guidelines for=20 guidelines.=E2=80=9D

Directors=20 also frequently singled out the timeliness of = recommendations=20 as an area for improvement. Directors offered comments = like=20 =E2=80=9CIt could have happened earlier=E2=80=9D and = =E2=80=9CIt could have been done=20 faster=E2=80=A6perhaps better with one person being = responsible for=20 keeping up the momentum.=E2=80=9D Recommendations were = sometimes=20 prepared as a =E2=80=9Ctechnical consultation=E2=80=9D = document as a way of=20 reducing both the amount of time needed to produce=20 recommendations and the level of expectations about = the rigour=20 of the process used. One director described a=20 recommendation-containing report that was: = =E2=80=9C=E2=80=A6 prepared as a=20 technical consultation document so it has a lower = status. They=20 should have been prepared by a study group and, even = better,=20 an expert committee. People have asked how can you say = a=20 technical consultation document is a WHO = recommendation, but=20 it has stood the test of time with other initiatives = coming to=20 similar conclusions. You can't develop a guideline in = less=20 than a year, but this doesn't work when there's = pressure.=20 Should there be a guideline for urgent = recommendations?=E2=80=9D

Several=20 directors identified the match between the resources = available=20 and the resources needed to develop recommendations, = and=20 attention to dissemination and implementation = strategies, as=20 other areas for improvement. Two directors identified = a lack=20 of resources as the problem. For example, one said: = =E2=80=9CWe had=20 inadequate time and resources. The recommendation was=20 developed during about 10 months. I believe this is = too short=20 a time. Would like to be able to use a more systematic = approach.=E2=80=9D Two other directors indicated that = the resources=20 needed to develop recommendations was the problem. For = example, one said: =E2=80=9CIt was a cumbersome and = resource-demanding=20 exercise.=E2=80=9D Several directors noted that = recommendations were=20 not being implemented after they were published. One = said: =E2=80=9CWe=20 published it, but just left it there.=E2=80=A6 The = recommendations=20 were never transformed into a programmatic approach. = It is a=20 common in-house failure to transform recommendations = into=20 action.=E2=80=9D Another said: =E2=80=9CThe marketing = of it, making people=20 aware, should have been thought of = earlier.=E2=80=9D

Directors=20 highlighted several other weaknesses with the = processes used=20 to develop recommendations, although most were = mentioned by=20 only one director. The weaknesses included a failure = to=20 involve key organisations, a failure to use evidence = from=20 other sectors, the creation of high expectations, a = conflict=20 over data, failure to use the guidelines for WHO = guidelines=20 (which were published after the process was started), = the=20 perceived need to choose between having a so-called=20 mega-meeting or using a smaller group to develop=20 recommendations, the failure to involve patients = sufficiently,=20 the failure to fit recommendations to health systems, = not=20 having had consultations earlier in the process, and = not=20 obtaining baseline data for an assessment.

The=20 anticipated benefits, potential harms, and costs of = adherence=20 to the recommendations were unevenly considered. All = directors=20 could cite one or more anticipated benefits of = adherence, such=20 as simplification of treatment, improved quality of = care,=20 better management of technologies, and reduced = morbidity and=20 mortality. Fewer directors could cite one or more = potential=20 harms of adherence. Indeed, several directors reported = that=20 there were no potential harms in adhering to their=20 departments' recommendations. For example, one = director=20 argued: =E2=80=9CNo harms are likely, since the = recommendations were=20 made by the top experts.=E2=80=9D Those directors who = could cite=20 potential harms provided general examples, such as=20 side-effects and the consequences of misapplication or = adaptation of the recommendations. One director = reported that=20 the potential harms were only considered implicitly in = the=20 discussion because it was feared that emphasising the = risks=20 might reduce the value of the recommendations, which = were=20 intended to help countries advocate for disease = control=20 programmes. Many directors identified both direct = costs and=20 opportunity costs associated with adhering to the=20 recommendations.

When=20 asked about how their department's recommendations = have been=20 used, directors provided examples such as educators = using them=20 in training programmes, WHO staff using them in their = work in=20 countries, and member states using them for = development of=20 policies. Several directors reported requests for = reports,=20 webpage hits, or translations of reports as indicators = of the=20 usefulness of the recommendations. Only a few = directors=20 reported any systematic monitoring of the uptake of = their=20 recommendations. Similarly, only a few reported = completed or=20 planned evaluations, which might partly be due to a = lack of=20 resources, as suggested by one director. =E2=80=9CWe = would love to do=20 it through a rigorous process. The problem is that = this would=20 require resources that we do not currently have and = cannot=20 reasonably expect in the foreseeable future.=E2=80=9D = Another reported=20 =E2=80=9Cno plans for evaluation because the cycle of = scientific=20 developments is so quick that it isn't = feasible.=E2=80=9D Most of the=20 reported evaluations were not rigorous assessments of = their=20 effectiveness. They included the collection of = indicators,=20 case studies, site visits, and feedback at = meetings.

The=20 four clinical guidelines that we examined did not = emphasise=20 evidence about effectiveness or processes that would = allow for=20 the explicit incorporation of other types of = information.24=E2=80=9327 Two of the reports = were called=20 =E2=80=9Cguidelines=E2=80=9D, one a =E2=80=9Ctechnical = consultation=E2=80=9D, and another a=20 description of =E2=80=9Cthe technical basis for the = guidelines=E2=80=9D. Two=20 of the reports stated that =E2=80=9CThis document is = not a formal=20 publication of WHO=E2=80=9D on the page containing the = publication=20 information. Three reports (including the supporting=20 documentation) did not contain a methods section. The = fourth=20 report contained a brief (less than one page) methods = section.=20 In all four reports the recommendations were neither = itemised=20 nor explicitly linked to evidence. All reports = included=20 references to primary studies or secondary sources. = Three=20 reports cited at least one systematic review as a = reference=20 (and at most four). The descriptions of the=20 recommendation-development processes used in = developing the=20 recommendations were brief and provided little = information=20 about group processes (panel 3).

Panel 3: Recommendation-development=20 processes used in four guidelines

The=20 following descriptions are taken from the four = guidelines=20 included in our document review.

=E2=80=A2=E2=80=9C=E2=80=A6year-long = process of=20 international consultative meetings in 2001, in which = more=20 than 200 clinicians, scientists, government = representatives,=20 representatives of civil society and people living = with=E2=80=A6from=20 more than 60 countries = participated.=E2=80=9D=E2=80=9CThe recommendations = included in this=20 publication reflect the best current practices based = on a=20 review of existing evidence. When the body of evidence = was not=20 conclusive, expert consensus was used as a basis for=20 recommendations.=E2=80=9D

=E2=80=A2=E2=80=9CThis = document was=20 prepared for the WHO =E2=80=A6by=E2=80=A6. The = document was reviewed by the=20 WHO Regional Advisors=E2=80=A6and approved by the WHO = Strategy and=20 Technical Advisory = Group=E2=80=A6=E2=80=9D

=E2=80=A2=E2=80=9CA WHO = Technical=20 Consultation on=E2=80=A6was held in Geneva, = Switzerland on 4 and 5=20 April 2001. Participants reflected a wide range of = expertise=20 in the document and use of=E2=80=A6drugs. =E2=80=9CThe = technical consultation=20 took the form of presentations based on working papers = and=20 plenary discussions, on the basis of which specific=20 conclusions and recommendations were agreed. The = proceedings=20 of the meeting and working papers form the basis of = this=20 report.=E2=80=9D

=E2=80=A2=E2=80=9CThe = guidelines=E2=80=A6are based=20 on both expert clinical opinion and research results. = A=20 technical review of existing programme guidelines was = carried=20 out with the cooperation of 12 WHO technical = programmes=20 through the WHO Working Group on=E2=80=A6. Some = modifications were=20 required=E2=80=A6. The draft guidelines were = subsequently reviewed in=20 several versions by clinicians and experts in specific = diseases who had experience in clinical and = public-health work=20 in developing countries, then examined in research = studies and=20 by field-testing the training = course.=E2=80=9D

=E2=80=A2=E2=80=9CSufficient data were not = available to make several guideline = decisions=E2=80=A6. Six studies=20 were carried out=E2=80=A6=E2=80=9D=E2=80=9CThe case=20 management charts and the modules were revised based = on this=20 experience and on the results of additional studies = and=20 analyses to help identify the best clinical=20 indicators=E2=80=A6=E2=80=9D=E2=80=9CThe revised=20 materials were made available to countries for closely = monitored = use=E2=80=A6=E2=80=9D

Several=20 directors indicated that there was a growing = recognition of=20 the need for more systematic and transparent = approaches to=20 developing recommendations and that there was progress = in this=20 direction. One director observed: =E2=80=9CThere has = been a culture=20 change, but there is room for improvement.=E2=80=9D = Another said: =E2=80=9CIt=20 is improving, but slowly. Many departments are doing = OK, while=20 others are not doing so well. Some have been too close = to=20 industry, often because of lack of resources.=E2=80=9D = A third=20 director also provided a long-range view: =E2=80=9CWe = are in the=20 middle of a process, which needs time. There is = increasing=20 understanding of the need for evidence-based guidance = and it=20 is becoming part of the WHO culture.=E2=80=9D

Back=20 to top

Discussion

The=20 guidelines for developing WHO guidelines do not seem = to be=20 closely followed when WHO develops recommendations for = member=20 states. For example, systematic reviews and concise = summaries=20 of findings (eg, balance sheets) are rarely used, = which means=20 that evidence is generally not retrieved, appraised,=20 synthesised, and interpreted using systematic and = transparent=20 methods. Processes for developing recommendations = typically=20 rely heavily on experts in a particular content area = and not=20 on representatives of those who will have to live with = the=20 recommendations or on experts in particular = methodological=20 areas (eg, information retrieval, systematic reviews, = economic=20 evaluations, and group facilitation). Although many = people we=20 spoke with viewed this as a problem, many others did = not.=20 Little attention seems to have been given by WHO to = how to=20 best help member states adapt global recommendations = or take=20 account of local needs, conditions, resources, costs, = and=20 values. Relatively little attention has also been = given by WHO=20 to roles and responsibilities related to effective=20 dissemination and implementation strategies and their = rigorous=20 evaluation.

The=20 strengths of our study include achieving a high = response rate=20 among the directors of a broad cross-section of WHO=20 departments, interviews that probed the contexts for = and=20 processes used in developing specific guidelines or = policies,=20 and augmenting the interviews with document reviews in = a=20 domain that could be expected to be a best-case = scenario=20 (developing clinical practice guidelines as opposed to = public=20 health or policy recommendations), and undertaking two = efforts=20 to verify our interpretations (sharing our written = summaries=20 of each interview with directors and sharing our = findings at=20 various forums within WHO). The verification process = yielded=20 only minor corrections. The study's weaknesses include = the=20 potential for social desirability bias, particularly = in terms=20 of identifying the use of evidence as an area for=20 improvement.

Although=20 the WHO guideline recommendations are consistent with = those=20 developed by other organisations,28=20 the actual processes used to develop recommendations = at WHO=20 seem to be less rigorous than those of others. None of = the=20 directors reported using the guidelines for WHO = guidelines and=20 only two reported plans to use them. Few directors = reported=20 using processes that were consistent with the = guidelines. An=20 unpublished in-house review, which was done before our = study=20 using the Appraisal of Guidelines Research and = Evaluation=20 (AGREE) appraisal instrument for assessing clinical=20 guidelines,14=20 noted that most WHO guidelines did not meet most of = the AGREE=20 criteria (Robin Gray, personal communication, 2003). = Reviews=20 of clinical practice guidelines produced by other=20 organisations also report that they often do not = adhere to=20 their own guideline recommendations.29=E2=80=9331 WHO also is not = alone in its failure=20 to recognise the danger of inadequately assessed = public health=20 and policy interventions, which, like clinical = interventions,=20 can also have unintended consequences.32=20 However, many organisations now report using = systematic and=20 transparent methods to develop clinical, public = health, and=20 policy recommendations, including a growing number of=20 organisations funded by government.33=E2=80=9335

Progress=20 in the way that WHO develops and disseminates = recommendations=20 for member states, and in how it supports member = states in=20 their efforts to adapt and implement recommendations, = will=20 require leadership. WHO's Cabinet recognised the need = for=20 using systematic and transparent methods to develop=20 recommendations when it endorsed the guidelines for = WHO=20 guidelines in 2003.23=20 Yet no mechanisms have been put in place to support = and=20 monitor adherence to the guidelines, and our study = suggests=20 that they are not being followed. Some directors = reported a=20 shift towards a culture that supports using systematic = and=20 transparent methods in developing recommendations, but = this=20 shift seemed to pertain more to clinical than policy=20 recommendations. WHO has not clearly articulated = whether and=20 how it will support member states in their efforts to = adapt=20 and implement recommendations.

Progress=20 will also require the resources for WHO to undertake=20 recommendation-development processes in a transparent = and=20 defensible way, and will need close attention to the = current=20 and emerging research related to these processes. All = of the=20 directors we interviewed were highly motivated and = trying hard=20 to do a good job. Many were frustrated by a lack of = resources=20 and feelings of being pressured by a lack of time and=20 perceptions of urgency. WHO relies heavily on external = financial support, so resources will probably have to = be=20 sourced from outside the organisation. However, WHO = could do=20 much better with the resources it has, both by setting = priorities and by adhering to its own guideline=20 recommendations. Considering that these guidelines = might be=20 most relevant to the development of clinical practice=20 guidelines and public health recommendations, future=20 iterations of the guidelines will need to incorporate = the=20 emerging research literature about developing policy=20 recommendations.12

Back=20 to top

Contributors

All=20 authors contributed to the design of the study, = participated=20 in interviews, contributed to the analysis and = drafting of the=20 report, and have read and approved the final version = of the=20 report.

Conflict of interest statement

AO=20 is a member of the WHO Advisory Committee on Health = Research.=20 JL is President of the PAHO/WHO Advisory Committee on = Health=20 Research, and a member of the Scientific and Technical = Advisory Committee of the Alliance for Health Policy = and=20 Systems Research, which is co-sponsored by and housed = within=20 WHO.

Acknowledgments

We=20 are grateful to all the people who agreed to be = interviewed=20 and thank them for taking time from their busy = schedules to=20 meet with us. We would also like to thank colleagues = at WHO=20 for their advice and comments on earlier versions of = this=20 report. This study was part of a project (http://www.practihc.org/) funded = by the=20 European Commission's 5th Framework International=20 Collaboration with Developing Countries, Research = (Contract=20 ICA4-CT-2001-10019). Lavis receives funding as the = Canada=20 Research Chair in Knowledge Transfer and = Exchange.

Back=20 to top

<= /DIV>

References

1. Murphy MK, Black NA, = Lamping DL, et al. Consensus development = methods, and=20 their use in clinical guideline = development. Health=20 Technol Assess 1998; 2: i-88. = CrossRef

2. Herrin J, = Etchason JA,=20 Kahan JP, Brook RH,=20 Ballard DJ. Effect of panel composition = on=20 physician ratings of appropriateness of abdominal = aortic=20 aneurysm surgery: elucidating differences between=20 multispecialty panel results and specialty society=20 recommendations. Health = Policy 1997; 42: 67-81. = Abstract | Full Text | Full-Te= xt=20 PDF (92 KB) | MEDLINE=20 | CrossRef

3. Ayanian JZ, = Landrum MB,=20 Normand SL, Guadagnoli E,=20 McNeil BJ. Rating the appropriateness = of coronary=20 angiography=E2=80=94do practicing physicians agree = with an expert=20 panel and with each other?. N Engl J=20 Med 1998; 338: 1896-1904. = MEDLINE

4. Fitch K, Lazaro P,=20 Aguilar MD, Martin Y,=20 Bernstein SJ. Physician recommendations = for coronary=20 revascularization. Variations by clinical=20 speciality. Eur J Public=20 Health 1999; 9: 181-217. = CrossRef

5. Vader JP, Porchet F, = Larequi-Lauber T, Dubois RW,=20 Burnand B. Appropriateness of surgery = for=20 sciatica: reliability of guidelines from expert=20 panels. Spine 2000; 25: 1831-1836. = MEDLINE=20 | CrossRef

6. Bernstein SJ, = Lazaro P,=20 Fitch K, Aguilar MD,=20 Kahan JP. Effect of specialty and = nationality on=20 panel judgments of the appropriateness of coronary=20 revascularization: a pilot study. Med=20 Care 2001; 39: 513-520. = MEDLINE

7. Devereaux PJ, = Anderson DR,=20 Gardner MJ, et al. Differences between = perspectives of=20 physicians and patients on anticoagulation in patients = with=20 atrial fibrillation: observational = study. BMJ 2001; 323: 1218-1222. = C