8. How Good is the Evidence? How NICE Makes Decisions

We may well have evidence-based advice, but how good is the evidence? There are grades of evidence and we must consider the reliability of the evidence on which we base decisions. This section will also discuss how NICE make their decisions.
This section will cover the following topics:

If you wish to go directly to one of the topics above, click on the blue underlined title

Not all evidence is equal. We shall look at grades of evidence or level of evidence and bias in studies. Health economics is mentioned again. There is insight into how NICE makes decisions.

What is the Evidence?

Authorities such as NICE often produce expert opinions based on the evidence available. How good is this evidence? Is it overwhelming or is it rather tenuous? Not all evidence is equal, and we need to know. If a person is to be judged on a decision that he or she has made, the judgement must be based on the evidence available to that person at that time. If a person does what was “best practice” at the time, that person must not be condemned in the light of future knowledge.

In law the verdict is guilty or not guilty. In medicine, there are much finer nuances

Evidence-based medicine is defined as “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients”. For the doctor with the patient in front of him the question is not about populations and studies but the individual whom he faces. In criminal law, a jury may spend hours or even days weighing up the evidence that has been presented before bringing a simple verdict of guilty or not guilty. In Scotland there is an additional option of not proven which in English law has to be given as not guilty.

In criminal law the onus of proof is beyond reasonable doubt whilst in civil law, the balance of probabilities will suffice. This means anything over a 50% chance. Hence it is much easier to prove a case in civil than in criminal law. Medicine also requires a diligent weighing and assessment of the evidence but the verdict tends to be more complex than simply effective or ineffective and the verdict is often given to reflect the weight of the evidence.

There are many questions that a reviewer has to ask:

  • What is the evidence?
  • How reliable is the methodology? Bad methodology is rarely obvious from reading the paper or the editor would not have published it. It usually comes to light some time later.
  • How convincing is the result? Perhaps the statistics were rather unimpressive or perhaps a small sample gave a large confidence interval. Was a confidence interval given?
  • Are there alternative explanations? Association and causation are not the same. We should always look for alternative explanations.
  • Is there selective publication? This is suggested by many papers showing unspectacular results and meta-analysis may be misleading without cut and fill.
  • Is there a conflict of interest? Just because a pharmaceutical company finances a study of its product does not mean that the paper is corrupt or invalid but the reader may be a little more wary.

Usually the evidence is not a single paper but many, perhaps with conflicting results. There are a number of pitfalls that the reviewer should remember.

  • Lack of evidence that something works should not be confused with evidence that it does not work. The study may be too small to form a firm conclusion. Similarly, lack of evidence of toxicity does not mean safety unless there have been extensive investigations to find it. Electronic cigarettes have not been around for long enough for studies of long-term safety.
  • Meta-analysis can be a useful tool, but it has some important limitations. The more data that is pooled, the less relevant it becomes to individual patients.1The limits of evidence-based medicine
  • The evidence-based approach may be more applicable to the use of drugs than to other treatment modalities, particularly holistic therapies. That is not to suggest that “complementary therapies” are not amenable to the scientific method.
  • Medical journals cannot always prevent papers from being “ghost written” by pharmaceutical companies.2Prevalence of articles with honorary authors and ghost authors in peer-reviewed medical journals This is where they pretend to come from an independent source but really they are not. There were a couple of classic papers which aimed to show that smoking had a protective effect against Alzheimer’s disease and Parkinson’s disease. A likely explanation for the findings is that both are diseases of old age and smokers are less likely to reach old age.3Cigarette smoking and Parkinson disease: the illusion of a neuroprotective effect. It transpired that they were both funded by the tobacco industry.
  • Case studies have been virtually replaced by evidence-based medicine. Case studies are just reports on individual patients rather than populations. An example is the notorious paper about the MMR vaccine and autism. In fact, the data had also been changed. Case studies are almost in the realms of anecdotes in terms of evidence.
  • A good doctor uses both individual clinical expertise and the best available external evidence, and neither alone is enough.

When faced with evidence there are three questions to ask:

  1. Is the evidence valid?
  2. Is it important?
  3. Is it applicable to the case in point? A doctor with a patient asks if it is applicable to the patient in front of him.

Levels of Evidence

Not all evidence is equal.
A hierarchy of evidence exists.

When expert advice is given, it is increasingly common for the review to give a grade to the level of evidence. This is an excellent idea but not all authorities use the same system. As well as the expected differences across the Atlantic, there are differences within the UK. However, these differences are small and there is reason to hope that in the near future a more uniform approach will be seen. There are two different ways that strength applies. One is the strength of the evidence. The other is the strength of the advice. The strength of the evidence should not be confused with the importance of the advice.

A “league table” of strength of evidence is called hierarchy of evidence and it is roughly as follows in descending order:

  1. Evidence from systematic reviews or meta-analysis of randomised controlled trials (RCTs)
  2. Evidence from at least one RCT
  3. Evidence from at least one controlled study without randomisation
  4. Evidence from at least one other type of quasi experimental type, such as a cohort study
  5. Evidence from non-experimental descriptive studies, such as comparative studies, correlation studies and case control studies
  6. Evidence from expert committee reports or opinions and/or clinical experience of respected authorities

The last is opinion where there is no good experimental evidence. It comes from those who are eminent in the field. Rather than evidence based medicine this is sometimes amusingly described as eminence based medicine.

Sometimes the strength of a recommendation is graded A to D, based loosely on the above. Strength, importance and relevance must not be taken as synonymous.

Is it Important?

Very rare events must be kept in proportion

The importance of a finding depends upon the significance of the event and the level of risk. A ten-fold increase in incidence may be ignored if it raises the risk from 1 in 50 million to 1 in 5 million. On the other hand, a 50% higher risk, raising the incidence from 4 in 10 to 6 in 10 is very important. The significance of the event is also important. Hence a 15% risk of a minor complication to a life-saving procedure is probably acceptable but if there was a 15% chance that a plane would crash, no one would travel by air.

It is a matter of putting everything into perspective.

Is it Relevant?

Look at the subjects in the original research and ask if they are comparable with your patient. A study on Australian aborigines of the outback may not be applicable to the commuters of Tunbridge Wells.

Is the research applicable to a wide range of people?

Two areas of concern are the management of heart failure and use of statins in the elderly. For good reasons of methodology most trials for heart failure used patients with no other medical conditions. Hence they tended to be between 55 and 70 years old. The typical patient who presents in the GP surgery is over 80 years old with multiple problems and on a variety of medication.

The large and original trials for statins to lower cholesterol usually had a maximum age for subjects of 75. They become increasingly more effective with increasing age and so, by extrapolation, it may be argued that they will be even more effective in the elderly. However, extrapolation may not be valid and until there were more trials in older people there was no direct evidence that statins helped people over 75. In both cases common sense and clinical judgement must prevail.

Since then the evidence for statins has been found.4Is it worth offering cardiovascular disease prevention to the elderly?

The management of heart failure in the elderly has also been reviewed.5Heart failure in very old adults

Osteoporosis is no more a disease of just women than heart disease affects only men.

All too often osteoporosis is seen as a disease of women although 15% of those affected are men. Many of the trials of drugs to treat osteoporosis, especially the bisphosphonates, have recruited only women. Therefore the doctor who faces a man with osteoporosis is very limited in the evidence for how to treat him. It may not be true that men and women respond in identical ways to therapy. A major cause of osteoporosis in women is the reduction in bone mass after the menopause and, in general, men do not have such a decline in their hormones. However, some men do. Some men have very low levels of sex hormones because of treatment for prostate cancer and the treatment is so good that they may have a reasonable quality of life for years. There are some reviews of the management of osteoporosis in men but the evidence is much less than for women.6Osteoporosis in men

Race may also be a relevant factor and the suggested management of hypertension (high blood pressure) varies according to race.7Nice. Hypertension in adults They also vary according to the age of the patient. This is all clear guidance with an evidence base.

Whilst strict inclusion and hence exclusion criteria are important to compare like with like for any trial, an excessive number of patients who are excluded leads to questions of the applicability of the outcomes to the general population. The problem of getting patients with heart failure and not a myriad of other diseases and treatments is easy to understand.

However, we may expect to find that there are plenty of young or middle-aged people with asthma without confounding pathologies. A study examined the number of people in a normal population who had asthma who would have been excluded from asthma trials.8External validity of randomised controlled trials in asthma: to whom do the results of the trials apply? It found that only about 6% of potential patients would have been recruited. These trials are the basis for evidence based advice to doctors. Very nearly 19 of every 20 patients that a doctor sees would fail the eligibility test on which the evidence is based to guide him on how to treat them.

Another potential problem is the use of surrogate outcomes. This means where a proxy is used for an outcome rather than the ultimate outcome. We know that high blood cholesterol and high blood pressure are risk factors for heart attacks and strokes. We also know that statins are good at reducing blood cholesterol and that there are several classes of drugs that are effective at lowering raised blood pressure. It may seem reasonable to believe that these effects would be translated into benefits in terms of reducing the incidence of heart attacks and strokes and deaths from those causes but it needs to be demonstrated, not assumed. A systematic review from Cambridge suggests that statins are effective in preventing coronary deaths in both primary and secondary prevention.9A systematic review and economic evaluation of statins for the prevention of coronary events.

Primary prevention is where there is no past history of the disease. Secondary prevention is where there is evidence of coronary disease such as a previous heart attack or angina. As expected, fewer patients need to be treated to prevent one death for secondary than for primary prevention. Numbers needed to treat (NNT) are discussed in Basic Maths in Medical Research and Decision Making. In fact, the effect of cholesterol lowering seems too good and there is evidence that statins may have an additional action and stabilise the atherosclerotic plaque to reduce coronary thrombosis too.10Stabilization of the vulnerable atherosclerotic plaque Trials that assess if statins reduce blood cholesterol can be very much smaller than those that look for outcomes such as heart attacks, strokes and mortality. The latter take longer to perform and are rather more expensive.

Is the research applicable to the patient?

Early studies on the effect of lowering blood pressure to prevent heart attacks and strokes with the older drugs were rather unconvincing. However, a more recent review has concluded that lowering blood pressure by 5 mmHg diastolic reduces the risk of stroke by an estimated 34% and coronary heart disease by 21% from any pre-treatment level.11Lowering blood pressure to prevent myocardial infarction and stroke

There is no threshold below which there is no benefit. However, not all classes of drugs are equal in terms of benefit per unit of blood pressure reduction and price is certainly no guide to efficacy. Chlorthalidone, a very cheap drug from the diuretic class was compared with doxazosin, a much more expensive drug in a study that included amlodipine, and lisinopril.12Major cardiovascular events in hypertensive patients randomized to doxazosin vs chlorthalidone

The chlorthalidone was equal or better than the doxazosin on some parameters and better on others. NICE gives guidance about the management of hypertension with regard to diagnosis, target blood pressures and drugs to be used.7NICE guidelines. Hypertension

After a heart attack there may be abnormal rhythms of the heart originating from the damaged muscle. This can lead to very abnormal patterns which even stop the heart and cause death. Therefore, suppression of these rhythms must be beneficial, or so a reasonable person may suppose. The theory does not always fit the practice. In the Cardiac Arrhythmia Suppression Trial (CAST) patients with abnormal rhythms were randomly allocated to one of three drugs to treat the abnormality or to placebo. The embarrassing result was that the mortality amongst those who received the active drugs was rather higher than amongst those allocated to placebo.13The Cardiac Arrhythmia Suppression Trial

It may seem wise to suppress abnormal rhythms after a heart attack but the evidence suggests otherwise.

The results were so marked that the trial was stopped prematurely as it was unethical to continue with a treatment that was causing so many excess deaths. The drugs had been licenced on the basis of their ability to suppress these abnormal beats called arrhythmias. Until then they had not been tested to see if they actually saved lives. Such drugs had been used so extensively on a false assumption that it was estimated that perhaps 100,000 extra deaths worldwide had resulted.

Because something seems sensible and logical, that does not mean that it is necessarily correct. We need evidence.

There is remarkably little regulation and evidence about the performance of hip and knee replacements

An area with a remarkable lack of evidence is the use of medical devices. This includes artificial joints that are used in operations such as hip and knee replacement. An editorial in the BMJ asks, 14Why aren’t medical devices regulated like drugs?” There are a vast number of hip prostheses from manufacturers all around the world and an orthopaedic surgeon should be able to ask, “What is the most appropriate for the patient in front of me?” but the answer is usually based on personal experience rather than evidence-based. In a paper from 2018 the authors state in the abstract, “We discuss the advances made in implant performance and the developments in perioperative practice that have reduced complications.

Assessment the outcomes of surgery should include patient-reported outcome measures and implant survival rates that are based on data from joint replacement registries. The high-profile failure of some widely used metal-on-metal prostheses has shown the shortcomings of the existing regulatory framework. We consider how proposed changes to the regulatory framework could influence safety.”15Hip replacement review Considering how many decades we have had hip replacements, it is astounding that this has not been done.

The former Committee on the Safety of Medicines (CSM) is now the Medicines and Healthcare products Regulatory Agency which regulates medicines, medical devices and blood components for transfusion in the UK. It does cover Medical devices regulation and safety too.

Medical devices are often much less researched than drugs

A paper from France noted that “Innovative high-risk medical devices, such as new types of heart valves or hip prostheses, become available on the European market more rapidly than in USA. This is due to the European legislation allowing early marketing of innovative high-risk medical devices before high-quality clinical evidence is obtained from randomized controlled trials.” It continued, “In contrast to the US, there is no requirement in Europe to demonstrate the clinical efficacy of high-risk devices in the pre-market phase. For the patient, this implies earlier access to innovative technology, but at the risk of potential safety issues.” The paper recommended limiting the introduction of novel high-risk devices with minimal clinical data.16Introduction of innovative high-risk medical devices in Europe

A term of questionable relevance in the literature is the “elderly primigravida”. A primigravida is a woman in her first pregnancy. An elderly primigravida is not a woman in her dotage who had an injudicious night on the sweet sherry but a woman with her first pregnancy who is more than 30 years old. The term “elderly” for one in her 30s does seem excessive. The evidence is good that they are at high risk. They are more likely to have complications of pregnancy and problems in delivery compared with younger mothers.17The obstetric course of elderly primigravidae

However, the epidemiology is based largely on figures from the 1950s to 1970s and the profile of the elderly primigravida has changed since then. She was typically a woman who had been trying to conceive for perhaps 10 years and now is successful for the first time in her 30s. Women who have difficulty getting pregnant are also more likely to have complications in pregnancy and delivery.

Women of higher social class are choosing to delay pregnancy and figures from the 1950s no longer apply.

The picture has changed. The elderly primigravida today is usually a woman who has used contraception with the intention of delaying starting her family for professional reasons. This means that she does not have a history of poor fertility and she is probably of upper social class. Higher social class puts her at lower risk compared with women of lower social classes. We need to rethink the concept of the elderly primigravida and to find a less derogatory term.

An acronym that is often used is POEM. It stands for Patient Orientated Evidence that Matters. We need more poets.

What is Proof?

Pilate asked Jesus, “What is truth?”

The question sounds rather like “What is truth?” and is just as contentious. The meaning of statistics and probability has been discussed. Even the most impressive P value (probability) is not absolute proof but suggests that it is extremely unlikely that there is no effect and the results were obtained by chance. To be able to understand the validity of evidence it is important to be able to understand possible shortcomings or common errors of methodology. The problem of bias from trials that were unpublished as they had negative results is too big and too important to ignore.

The various forms of research have already been discussed. They include Randomised Controlled Trials, which includes meta-analysis, Cohort or Longitudinal and Epidemiological Studies and Qualitative Research.

In 2015, the BMJ ran a poll asking readers to nominate which paper they should be most proud of publishing over the last 20 years. The poll attracted 576 votes across 55 countries. Professor Doug Altman’s editorial,18The scandal of poor medical research was the clear winner with attracting 252 votes, or nearly 44% of the total. He asked: “What should we think about researchers who use the wrong techniques, use the right techniques wrongly, misinterpret their results, report their results selectively, cite the literature selectively, and draw unjustified conclusions? We should be appalled. Yet numerous studies have shown that all of the above phenomena are common. This is surely a scandal.” It starts, ” We need less research, better research, and research done for the right reasons.”

A paper from 2005 was called 19Why Most Published Research Findings Are False. It berates poor methodology and conclusions that are not justified by the evidence. Do not believe that because something has appeared in a scientific journal and even been peer reviewed that it is incontrovertible truth. If we cannot rely on the literature, it is even more difficult to decide what is truth.

Association and Causation

This was also covered in the section on Cohort or Longitudinal and Epidemiological Studies and there is much more about it in that section. Because two things are in some way linked does not necessarily means that one causes the other. It is worth noting the key principles for any scientific study that seeks to prove causation rather than merely association, as set out by the great epidemiologist Sir Austin Bradford Hill in 1965:

  • Is there evidence from true experiments in humans?
  • Is the association strong?
  • Is the association consistent from study to study?
  • Is the temporal (time) relationship appropriate? Did the postulated cause precede the postulated effect? Otherwise it is less likely to be the cause than the effect.
  • Is there a dose-response gradient? The more cigarettes are smoked per day the greater the risk of lung cancer.
  • Does the association make epidemiological sense? A link between passive smoking and breast cancer is nonsensical when there is no link between active smoking and breast cancer.
  • Does the association make biological sense? Look at ice cream and drowning.
  • Is the association specific?
  • Is the association analogous to a previously proven causal association such as a virus causing cancer?

Most of the weird studies that newspapers like to report do very badly when assessed like this.

Health Economics and Effective Health Care

Evidence-based medicine asks simply if an intervention is effective, but we cannot ignore such matters as cost and value. One drug may be marginally more effective than another but vastly more expensive. Should doctors change all their patients to the more effective drug or only a few in special cases? Of course, there will be the usual headlines “Cost-cutting put ahead of patient care” but we must live in a real world with a finite budget from the taxpayer.

Quality of life is very important

Quality of life issues have also been discussed. It is another example of how “soft criteria” may be given numerical value. The management of small and pre-term infants has resulted in the salvage of many who would formerly have died but is the effort justified in terms of long-term handicap?

Modern medicine provides ethics with many challenges. Value is a judgement that we cannot ignore. There is much more on calculating matters related to cost and effectiveness and quality of life in the chapter Basic Maths in Medical Research and Decision Making

Bias in Studies

As we have seen, bias in studies can be a problem when we are seeking the truth. At an early stage we have seen that a scientist does not ask “How do I prove this?” Instead he asks an open question of “How can I test my hypothesis?” A common source of bias is called confirmation bias. This means that people set out to prove their point. If they do so they publish. If not, they do not publish and yet it is as important to know what does not work as what does. As well as biasing publication this can also bias results.

When the Cochrane Organisations scours the literature to give an authoritative opinion they have to take into account likely sources of bias in research. They have developed a tool to help assess randomised controlled trials for sources of bias. I shall not go into what this involves but it is an important step.20The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials The interested reader may care to follow that reference for a free article from the BMJ. Basically, they aim to find that if a group of trials are selected for meta-analysis that there is not some inbuilt bias that may distort the final picture.

A retrospective look at more than 1,000 RCTs approved by research ethics committees in Canada, Germany or Switzerland between 2000 and 2003 found that a quarter of them were abandoned, and that nearly a half (44%) were never published as full journal articles.21Discontinued Randomized Trials

Trials that are funded by the investigator are more likely to fail than trials funded by industry. The commonest reason for discontinuation is lack of recruitment. At least they are not admitting that the reason for not publishing is not getting the “right” result.

The true source of an article may not be apparent

Often we may believe that evidence for advice is very much stronger than it really is. Bias in studies is a common cause. Another problem is that firm quantitative advice does not have a strong foundation but experts have been put under pressure to give guidance based on numbers. We all know about eating five portions of fruit and vegetables a day. The evidence that fruit and vegetables should be an essential part of our diet its strong but there has been dispute as to whether the number of portions should be five, seven or even ten a day. Quite simply, it is not possible to put a firm recommendation on the numbers but they are there as a form of guidance.

How much alcohol is it safe to drink?

How much alcohol is it safe to drink? We are all different and where it becomes a problem varies, and diet and genetics are probably also important. This makes it impossible for guidance to be universally applicable. The first guidelines were for men to limit consumption to 28 units a week with 21 units for women. This rather lower figure for women than men may seem rather greater than expected, but women do seem to be more susceptible to the adverse influences of alcohol than men and they usually have a lower lean body mass. The recommendations were later reduced to 21 units for men and 14 for women but now the figure is 14 units a week for both women and men. I suspected that the new gender equality may be driven by political correctness rather than biology. However, the chief medical officer for England says that it is related to new evidence about the risk of cancer with alcohol consumption.

High alcohol consumption in pregnancy can produce fetal alcohol syndrome in the baby. A smaller amount can produce fetal alcohol effects. There is no obvious threshold and so most advice is to abstain for the duration of pregnancy. This seems wise but the woman who has an occasional light drink in pregnancy should not feel unduly guilty.

Confirmation Bias

What we see depends in our perspective

People are easily convinced by arguments or results that confirm their existing ideas or opinions, regardless of how tenuous the argument or evidence may be. On the other hand, it is most difficult to convince anyone of something that is contrary to their beliefs, no matter how cogent the evidence. This is called confirmation bias. It is easy to see how politicians cherry-pick the evidence or statistics that match their ideas whilst dismissing anything to the contrary. Donald Trump is by no means unique in this way.

If a dozen people see a road traffic accident and the police interview them all independently there will be 12 different stories, some very different from others. At a football match a player goes forward to attack. Was he offside? Everyone will have a clear and firm opinion based solely on which side they support. People sometimes say, “I don’t care what anyone else says…” which is a startling admission of a closed mind unwilling even to consider any evidence or argument that is contrary to prejudice.

Scientists should be objective, but they are human and exhibit human frailties. We tend to read newspapers with a political slant that accords with our own and we tend to mix with people who have similar opinions and outlooks to our own. This is called homologous sorting. Therefore, people with minority (usually extremist) views believe that their opinion is shared by the majority.

Confirmation bias has led to dramatic failures to foresee disasters that are about to happen when the evidence is strong, but it runs counter to the thoughts and wishes of the one who makes decisions. The Japanese attack on Pearl Harbour in 1941 should have been foreseen and the Soviet Union had plenty of evidence that Germany would attack them before it happened. The film “A Bridge Too Far” is the story of Operation Market Garden in 1944 to land allied troops in the occupied Netherlands. Dutch intelligence reported two Panzer divisions in the area, but they were dismissed as incorrect or the tanks were not operational simply because it was inconvenient to the ideas of those in charge. The result was disastrous. Max Hastings lays this out in this book The Secret War: Spies, Codes and Guerrillas 1939-1945. He says that both Hitler and Stalin were so averse to hearing bad news that it was kept from them. Fortunately, this probably contributed to some bad decisions that they made. He also commends British intelligence for being honest and unbiased in its reporting. This does not mean that it was always right, but it gave it a distinct advantage over others.22(Max Hastings. The Secret War: Spies, Codes and Guerrillas 1939–1945.)

The journal that receives the submission will not know how selective and biased it is.

As we have seen in the section on randomised controlled trials, it is important that the researchers decide in advance how they will analyse their data. However, it seems that the choice of analytical method may depend on what they are seeking to show. A report of a paper in 2018 by Medscape had the headline “The Year’s Most Important Study Adds to Uncertainty in Science.” I have not given a reference as it requires a password for access. The paper being discussed was “Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results.”23Many analysts, one data set

They gave 29 teams involving 61 analysts the same data set to address the same research question which was whether soccer referees are more likely to give red cards to dark-skin players than to light-skin players. The nature of the question is unimportant, but the result is. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Subjective, analytic choices influence research results.

It is interesting to see how opinion has changed over the years about the effects of hormone replacement therapy for women (HRT) on the risk for coronary heart disease and stroke. A review of 1992 in the British Medical Bulletin was positive about its protective effect.24.Hormone replacement therapy and cardiovascular disease An American review of the same year claimed that mortality from coronary heart disease decreased 20% to 40% in women on HRT.25Hormone Replacement Therapy A much later paper from 2017 was called 26The evidence base for HRT: what can we believe? Current wisdom is that HRT does not protect the heart and circulation and may even increase risk.


Are guidelines best practice or “painting by numbers”?

Guidelines have become more and more part of clinical practice, but they have their advocates and detractors. This reference is from 1999 but still applicable.27Potential benefits, limitations, and harms of clinical guidelines. To some, they represent best evidence-based practice and hence anything that departs from them is not best practice and so is unacceptable. To others they are “painting by numbers” and a one-size-fits all. In a workplace that sees far more litigation and complaints, doctors may feel safer abiding by the rules as they represent a “safety-net”. In February 2019, a commentator in the BMJ said that when faced with guidelines he asks:

  • Does the guideline apply to my patient?
  • Is the advice pragmatic?
  • Is it what my patient wants?

One of the main problems with guidelines is that they are only as good as the evidence and the people who laid down the guidelines. We have seen how the management of heart failure is based on patients far younger than those who usually present with the condition and there is very little evidence on the management of osteoporosis in men although they represent 15% of sufferers. Like them or loathe them, guidelines are with us. It is said that rules are for the guidance of wise men and the obedience of fools. Here, “fools” might not mean one who is stupid but one who lacks knowledge and expertise in the field. Here he should obey the rules. The one who is more knowledgeable of the subject understands the limitations of the guidelines and knows how he can improve on them for the benefit of the patient in front of him.

Medicinal Cannabis

A contentious area has become the use of cannabis for several medical conditions. This is totally separate from any debate about the legal status of recreational cannabis just as the use of opioids and even cocaine in medical practice is unrelated to the illicit use of heroin or crack cocaine. At the time of writing in 2019, there has been great pressure from pressure groups and individuals to get the NHS to provide medicinal cannabis but there are two problems. One is that evidence of efficacy and lack of toxicity is not strong enough to permit a product licence. The other is the price. In the chapter Basic Maths in Medical Research and Decision Making we saw how NICE use quality of life measures such as the QALY to add to price to see if it gave value. I have read of cannabis for the control of refractory epilepsy costing £3,000 a month. Even with a marked improvement in quality of life it would not offer as much as 1.0 QALY per year but at a cost of £36,000 per year. However, the NHS could probably negotiate a more realistic price.

A review by NHS England found a lack of good quality randomised control trial data.28NHS England Barriers to accessing cannabis-based products for medicinal use Hence, they were unable to recommend cannabis-based products for medicinal use. The report said the lack of evidence gap was most marked for products containing delta-9-tetrahydrocannabinol (THC). There was better quality data for products containing cannabidiol (CBD) and clinicians were more willing to prescribe these for some conditions. However, clinicians interviewed for the review said more robust evidence demonstrating adequate safety of cannabis-based products for medicinal use (CBPMs), and information on their clinical cost-effectiveness, were needed.

The NHS review recommended that research into the efficacy of CBPMs was a priority. This should include research into five areas. These were:

  • CBD as an add on treatment for adult patients with fibromyalgia or persistent treatment-resistant neuropathic pain
  • CBPMs for chronic pain in children and young people
  • CBPMs for people with spasticity
  • CBD for severe treatment-resistant epilepsy in children, young people, and adults

NICE issued draft guidance on CBMPs. It did not recommend the use of an oromucosal spray containing CBD and THC (trade name Sativex) for treating spasticity in people with multiple sclerosis because, despite evidence of benefit, it was not cost-effective at its current price.

Epilepsy Action said that it was unacceptable for patients to have to wait many years for trials to be completed. Without trials there is no evidence. I would be most unhappy if “popularism” led to the release of new drugs without proper testing to meet public demand. I wonder if a limited release on a named-patient basis may be a good interim compromise if the price is acceptable. Drugs for serious and fatal diseases such as Ebola and AIDS have been fast-tracked because the need was pressing.

There is rather more about medicinal cannabis in the chapter Other Natural Products. I put it there, rather than in Herbal Remedies partly because that chapter is so full and partly because medicinal cannabis is so removed from the natural product that it hardly classifies as a herbal remedy.

Decision Making

Do we accept something or do we reject it? This depends upon a number of factors. How important is the benefit to be gained? What is the chance of an adverse event (something going wrong with a bad outcome)? How bad is the bad outcome?

Charles Blondin crosses the Niagara Falls with a man on his back

If I ask you to walk along a beam that is 10cms wide and 15cms above the ground you would probably regard this as easy. However, if the same beam was 2 metres off the ground you would probably be less sure. If it was 20 metres above the ground you would probably not do it. What is the difference? It is the same beam with the same width and length. The difference lies in the nature of the adverse event. If you fall off a beam that is 15cms above the ground you are unlikely to come to any harm. If you misplace your step 2 metres off the ground the fall may cause injury. The same event with the beam 20 metres above the ground would probably be fatal. The chance of an adverse event has not changed but the severity of that event has. If I offered you £1,000 to do it you would be more likely to give it a try, at least at 2 metres if not at 20 metres.

  • How substantial is the benefit?
  • What is the risk of an adverse event?
  • How bad it an adverse outcome?

The benefit, the degree of risk and the nature of that risk are all involved.

What is an acceptable risk profile for a drug? This can be examined in a similar way. What is the benefit? If it is the possibility of curing a fatal disease such as cancer or AIDS then a higher risk of a significant adverse effect is much more acceptable than if a less serious illness is to be treated. Contraceptives are given to healthy young women and so the risk of serious adverse effects must be very small. The same is true of immunisations and treatment for acne. How likely is this adverse reaction to occur? Even a serious adverse event may be acceptable if the likelihood is very low. A mild and transient rash or diarrhoea for a few days may be acceptable where kidney failure would not.

Astroturfing is creating synthetic “grass-roots” opinion

Some of the issues that agencies such as NICE and the regulatory agencies have to consider are discussed in the next chapter. They may be under pressure to approve or recommend drugs where the evidence base for final outcomes rather than surrogates, are either not available or are unimpressive. Numbers needed to treat for one to benefit and value for money are possibly also absent or of poor quality. An outcry may be orchestrated by a patient support group although I often suspect that much has been fanned and encouraged by the relevant pharmaceutical company. In politics a synthetic “grass roots” opinion is called Astro-turf. Politicians join the fray. They rarely have much understanding of science but they fully comprehend the need to be re-elected. Drugs have been fast-tracked including the early drugs for AIDS and treatments for Ebola. Where there is no effective alternative and the disease has a horrific mortality such risks can be justified. Otherwise we need to have decisions based on full evidence that has been dispassionately assessed rather than mob rule.

The thinking and analysis here can be used outside of medicine too. How does a person who is tempted, decide whether or not to commit a crime? Logically, we may expect there to be an assessment of the benefit to be had, the risk of getting caught and the result of getting caught. The evidence suggests that the perceived risk of getting caught is far more important than the outcome. Hence, stiffer sentences are little deterrent if the person thinks that the risk of being caught is remote. Therefore, improving detection, whether this is real of perceived, is far more effective than draconian sentences or more inhumane conditions in prison. Most of us tend to be rather optimistic and do not expect to get caught.

Further Resources


  1. Tonelli MR; The limits of evidence-based medicine. Respir Care. 2001 Dec;46(12):1435-40; discussion 1440-1
  2. Flanagin A, Carey LA, Fontanarosa PB, et al; Prevalence of articles with honorary authors and ghost authors in peer-reviewed medical journals.;JAMA. 1998 Jul 15;280(3):222-4.
  3. Riggs JE Cigarette smoking and Parkinson disease: the illusion of a neuroprotective effect. Clin Neuropharmacol. 1992 Apr;15(2):88-99.
  4. Gohlke H. Is it worth offering cardiovascular disease prevention to the elderly? Eur J Prev Cardiol. 2013 Feb;20(1):121-6.
  5. Forman DE, Ahmed A, Fleg JL. Heart failure in very old adults. Curr Heart Fail Rep. 2013 Dec;10(4):387-400.
  6. Banu J. Causes, consequences, and treatment of osteoporosis in men. Drug Des Devel Ther. 2013 Aug
  7. Nice. Hypertension in adults: diagnosis and Hypertension in adults: diagnosis and management. August 2011. Updated 2016.
  8. Travers J, Marsh S, Williams M, Weatherall M, Caldwell B, Shirtcliffe P, et al. External validity of randomised controlled trials in asthma: to whom do the results of the trials apply? Thorax. Mar 2007; 62(3): 219–223. [full text]
  9. Ward S, Lloyd Jones M, Pandor A, Holmes M, Ara R, Ryan A, Yeo W, Payne N. A systematic review and economic evaluation of statins for the prevention of coronary events. Health Technol Assess. 2007. Apr;11(14):1-160, iii-iv. [full text]
  10. Akdim F, van Leuven SI, Kastelein JJ, Stroes ES. Pleiotropic effects of statins: stabilization of the vulnerable atherosclerotic plaque? Curr Pharm Des. 2007;13(10):1003-12.
  11. Law M, Wald N, Morris J. Lowering blood pressure to prevent myocardial infarction and stroke: a new preventive strategy. Health Technol Assess. 2003;7(31):1-94.[full text]
  12. ALLHAT Research Collaborative Group. Major cardiovascular events in hypertensive patients randomized to doxazosin vs chlorthalidone: the antihypertensive and lipid-lowering treatment to prevent heart attack trial (ALLHAT). JAMA. 2000 Apr;283(15):1967-1975 [full text]
  13. Echt DS, Liebson PR, Mitchell LB, Peters RW, Obias-Manno D, Barker AH, Arensberg D, et al. Mortality and morbidity in patients receiving encainide, flecainide, or placebo. The Cardiac Arrhythmia Suppression Trial N Engl J Med. 1991 Mar 21;324(12):781-8. [full text]
  14. Godlee F. Why aren’t medical devices regulated like drugs? BMJ 2018;363:k5032. 29th November 2018.
  15. Ferguson RJ, Palmer AJ, Taylor A, Porter ML, Malchau H, Glyn-Jones S. Hip replacement. Lancet. 2018 Nov 3;392(10158):1662-1671.
  16. Stordeur S1, Vinck I, Neyt M, Van Brabandt H, Hulstaert F. Introduction of innovative high-risk medical devices in Europe: are clinical efficacy and safety guaranteed? Rev Epidemiol Sante Publique. 2013 Apr;61(2):105-10
  17. Love RM. The obstetric course of elderly primigravidae–a review. J R Nav Med Serv. 1992 Winter;78(3):159-63.
  18. Altman DG. The scandal of poor medical research BMJ 1994;308:283
  19. Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124
  20. Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA; Cochrane Bias Methods Group; Cochrane Statistical Methods Group. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011; 343:d5928 [full text]
  21. Kasenda B, von Elm E, You J, Blümle A, Tomonaga Y, Saccilotto R, et al. Prevalence, Characteristics, and Publication of Discontinued Randomized Trials. JAMA. 2014 Mar 12;311(10):1045-51. [full text]
  22. Max Hastings. The Secret War: Spies, Codes and Guerrillas 1939–1945. 2015. Hardback, paperback & Kindle.
  23. Silberzahn R, Uhlmann EL, Martin DP, et al. Many analysts, one data set: making transparent how variations in analytic choices affect results. Adv Methods Pract Psychol Sci.
  24. Meade TW, Berra A. Hormone replacement therapy and cardiovascular disease. Br Med Bull. 1992 Apr;48(2):276-308.
  25. Nachtigall LE, Nachtigall MJ. Hormone Replacement Therapy. Curr Opin Obstet Gynecol. 1992 Dec;4(6):907-13.
  26. Lange RD. The evidence base for HRT: what can we believe? Climacteric. 2017 Apr;20(2):91-96.
  27. Woolf SH, Grol R, Hutchinson A, Eccles M, Grinshaw J. Potential benefits, limitations, and harms of clinical guidelines. BMJ 1999;318:527. [full text]
  28. NHS England Barriers to accessing cannabis-based products for medicinal use on NHS prescription Findings and Recommendations. Gateway Reference number: 000842, August 2019

Site Index

This website is now completed, although I shall continue to do updates. The following list shows the sections or chapters. Just click on the topic in blue to go to that part of the site.

1 Introduction
2 A Very Brief History of Science And Medicine
Fundamentals of Medical Science
3 Finding Good Medical Advice and Evidence Based Medicine
4 Randomised Controlled Trials
5 Cohort or Longitudinal and Epidemiological Studies
6 Qualitative Research
7 Basic Maths in Medical Research and Decision Making
8 How Good is the Evidence?
9 Ethics in Practice and Research
Public Health Issues
10 Screening Programmes
11 Fake News and Vaccine Scares
12 Electronic Cigarettes (E-Cigarettes)
13 Motor Vehicle Emissions, Air Pollution and Health
14 COVID-19. What You Need to Know
15 Who is at Risk from COVID-19
16 What we Must Learn from the COVID-19 Pandemic
17 Basics of Nutrition
18 Exercise, Obesity and Diets for Weight Loss
19 Diets and Nutrition for Health and Fitness
20 Supplements
Complementary and Alternative Medicine
21 Introduction to Alternative Healthcare
22 Homeopathy
23 Acupuncture
24 Manipulation of the Spine
25 Reflexology
26 Herbal Remedies
27 Other Natural Products
28 Chelation Therapy
29 Hypnosis
30 Other Modalities of Complementary and Alternative Medicine
Some Controversial Diseases
31 Fibromyalgia
32 Chronic Fatigue Syndrome (CFS) or Myalgic Encephalitis (ME)
33 Systemic Candidiasis and Leaky Gut Syndrome
34 Mobile Phones, Masts, Wi-Fi and Electro-sensitivity
The Environment
35 Global Warming and Climate Change
36 Alternative Energy
Some Final Thoughts
37 Still Searching for the Age of Reason