The Problem With So-Called Evidence-Based Policy

How do we follow the evidence to make purely political decisions?  The authors describe two types of evidence: off-the-shelf evidence and pragmatic evidence.

Off-the-shelf evidence, according to the authors, are available information provided by scientific knowledge.  But that information is often contradictory and usually is not directly applicable to the current situation.  In the case of the SARS-CoV-2 virus there wasn't off-the-shelf evidence.  Available off-the-shelf evidence for the SARS-CoV-1 virus (SARS and MERS viruses) indicated that a pandemic was unlikely.  The scramble to obtain evidence specific to the SARS-CoV-2 was clouded with a great deal of uncertainty; was the COVID outbreak similar to the SARS and MERS outbreaks or not?  Off-the-shelf evidence could not answer that question.

Pragmatic evidence, according to the authors, relates to the context of the decisions that must be made.  At one time, pragmatic evidence was called 'common sense'.  The common sense approach to the situation was to avoid over reacting.  While there had been isolated outbreaks early on, those outbreaks were not inconsistent with off-the-shelf evidence for the SARS and MERS outbreaks.  The isolated outbreaks had not coalesced into a pandemic scale situation requiring immediate action.

There is also a third type of evidence that the authors have ignored but are utilizing to write the article.  That third type is the evidence of hindsight.  No effort is required to obtain the evidence of hindsight other than allowing time to pass.  Ostensibly the evidence of hindsight becomes both off-the-shelf evidence and pragmatic evidence for future situations.  But it isn't possible to rewind the clock and apply the evidence of hindsight.  The evidence of hindsight is only useful for politically identifying mistakes in decision making and assigning blame.

The evidence of hindsight is used to argue that decision makers should have known.  But that argument deliberately ignores that the evidence available when the decisions were made has only been clarified by hindsight.  And no effort was required to obtain hindsight other than to sit back and allow time to pass.  The evidence of hindsight only allows those who have done nothing to complain about those who were expected to do something.  

Evidence-based policy is meaningless from a practical standpoint.  But evidence-based policy isn't about practicality.  Evidence-based policy is really about shifting blame onto others.  Evidence-based policy is really about politically removing responsibility for inaction from those who simply sit back and wait for time to pass.

S E E D E D   C O N T E N T

"Evidence-based policy" has become a mantra, a condition of rational action, a bright line that separates decisions that have science and facts on their side from those that are merely political, emotional, or irrational in motivation. The COVID-19 pandemic is proving no exception, as claims that policy decisions must be based on science and evidence are rampant. On one level such claims are entirely obvious; of course we want evidence to guide our response. Accordingly, governments make reference to scientific evidence (as well as expert judgment) to justify their actions (or, more often, inaction) and to explain their changes in policy as responding to changes in the available evidence. But on another level, the claim of evidence-based policy is a shield behind which politics are hidden and responsibility is evaded.

To understand the role of "evidence" in policy-making, especially during times of crisis, it is worth first considering the concept of evidence itself. The ideal of evidence-based policy can be traced back to the technocracy movement of the beginning of the twentieth century, but it has been most fully articulated and implemented through the aspirations of evidence-based medicine.

Evidence-based medicine depends on the recognition of hierarchies of scientific evidence in medical practice, with randomized controlled trials widely considered to be the "gold standard" of evidence in support of clinical decisions as diverse as prescribing drugs, choosing surgical procedures, and recommending diets or various therapies.

Even in medicine, however, this ideal is inherently problematic. A doctor has access to the results of randomized, controlled trials about a drug or procedure, but must still make judgments in light of considerations that include the uncertainties revealed by the trials, the particulars of any given patient (how, for example, might the drug interact with other treatments the patient is receiving?), and even the cost of the treatment.

Two types of evidence

What type and amount of information are sufficient evidence for taking action? This is not something that comes prepackaged or predetermined. What is considered evidence, and how evidence is considered, depends on many factors. One way to think about evidence is to divide it into two general groups, which we will call "off-the-shelf" and "pragmatic." Off-the-shelf evidence may come from formal scientific studies whose results were generated in the context of an experiment such as a randomized, controlled trial. In the case of COVID-19, it mostly comes as observational data (about rates of infection, number of deaths, and so on) whose implications must be extrapolated from one context to another. We may somewhat simplistically understand such evidence as prepackaged facts, available for deployment on behalf of decisions.

Pragmatic evidence, in contrast, relates directly to the context of the decision, and is particularly relevant not only in medicine (as in the case of needing judgment in treating patients) but also in fields of practice such as engineering or policy decision-making, where important decisions involve judgments that often mix off-the-shelf facts with considerations about desired outcomes.

The importance of this distinction is on display in the decisions that European countries made about responding to COVID-19, as off-the-shelf evidence was first becoming available. What we can see now is that although different countries made different decisions leading to different consequences, overall most governments delayed action to the point where controlling the pandemic's negative outcomes for public health and the economy became very difficult.

Various European governments have justified their delay in adopting strict preventive measures by suggesting that at the time (mid- to late-February or the beginning of March) the available evidence did not justify such measures. Yet today we can be fairly sure that an earlier enactment of certain preventive actions would have forestalled some of the deaths due to the virus, as well as slowed contagion and—up to a point—reduced the economic fallout. An important question, then, is whether, as governments now claim, their earlier inaction was justified by the available evidence.

Most governments delayed action to the point where controlling the pandemic's negative outcomes for public health and the economy became very difficult.

In the context of the current crisis, what's at stake is whether governments acted "rationally"—with due deference to evidence—and therefore should not be held accountable by their citizens for the consequences of their inaction. The most significant point about this argument is that—moral, political, or legal issues aside—it flows at least implicitly from a particular interpretation of what counts as relevant evidence for decision-making. We want to illustrate this by a few examples.

How much evidence is enough?

Throughout most of February, many European governments stated that what was happening in China did not constitute sufficient evidence for taking action in their countries. Even when the coronavirus surged in Italy, several governments (including in the United Kingdom, France, and Spain) still did not consider the events in Italy as sufficient evidence for action. In other words, they did not consider that there was sufficient evidence to predict that only a few days later their own countries would be facing a similar situation, and that taking (at least some) preventive action might therefore be warranted.

To take just a few examples: on February 12, the organizers of the Barcelona Mobile World Congress decided to cancel the 2020 conference, after several global telecommunication companies had announced they would not participate because of the crisis. The Spanish health minister, however, declared that there were no public health reasons that warranted canceling the event because in Spain there did not exist on that day any public health crisis.

The European Centre for Disease Prevention and Control, on February 18, said the risk of COVID-19 spreading throughout the European Union was "low to moderate." When it issued this assessment, there were already confirmed cases in 19 member states (out of 26). The European Commission issued a recommendation for not closing any of the internal borders between EU members.

On March 15, France held its first round of municipal elections. This meant opening 70,000 polling stations across the country, to serve a total of 48 million voters. The French government stated that its scientific committee had agreed that citizens participating in the elections would not face a higher risk than going shopping. Whether or not that was the case, the fact that the elections posed an additional risk (on top of other activities, such as going shopping) apparently did not play any role in the assessment.

A high-ranking official at ASISA, an important private insurance and health provider, in an interview stated: "Which government, without deaths [due to the virus], would have gone along with confinement, and crash the economy of a country?" In our interpretation, this means that the deaths in their own country were the evidence that public authorities needed in order to be able to act. Evidence (deaths) from other countries was simply insufficient to "crash the economy."

What the political decision-makers in these cases took (consciously or unconsciously) for granted is that what was happening in China did not constitute relevant or sufficient evidence, and that the only relevant evidence would be the existence of a sufficient number of confirmed cases of infection within their own country. In other words, the health authorities were saying that there was no reason to take preventive action unless the pandemic was actually having a direct and visible impact on their own citizens.

The existence of an unknown but certainly important number of asymptomatic cases also made it impossible to really understand the magnitude of the contagion. Moreover, it is certainly possible to argue that the events in China, where most of the available off-the-shelf-evidence came from in the early weeks of the pandemic, did not constitute sufficient evidence for taking action in Europe because it was not possible to know with any certainty how to extrapolate such data to the European context.

Evidence for inaction

Yet there did exist extrapolatable data that could support inaction: the effect of the 2002-2004 SARS epidemic—caused by the novel coronavirus SARS-CoV, later renamed SARS-CoV-1—on Europe had been negligible. Furthermore, there even existed a pragmatic consideration in favor of government inaction in Europe: the case of the 2009-2010 influenza A virus (H1N1). A number of countries ultimately disposed of their vaccines for this virus because in the end it proved much less virulent than originally thought. In fact, in this case the World Health Organization was criticized for having erred in its predictions and being alarmist.

Is the ideal of evidence-based decision-making a cover that allowed decision-makers to weight their own political risk above the possible societal consequences of inaction?

Our point here is that in light of the events in China, European governments could have adopted measures (e.g., social distancing, banning large events, mobilizing massive testing capacity, and purchasing necessary protective gear) even in the absence of confirmed local cases. But the early imposition of such measures could not have been based solely on off-the-shelf evidence. Rather, making such decisions earlier, before there was evidence of significant local incidence of COVID-19, would have to have been based on purely pragmatic evidence applied to judgments about the possible impact of the coronavirus on public health, as well as on the economic consequences of the severe social and economic measures that did eventually turn out to be necessary (and that had already been implemented in Hubei province, China, as was well known). Decision-makers also had to take into account whether they had the political support to impose such measures—another matter where the relevant evidence was necessarily pragmatic.

The evolution of this current coronavirus could have been more like what happened with influenza A, in which case the economic costs of early adoption of preventive measures would have been unnecessary. And there would have been political costs, too, because European governments would have been accused of alarmism, yet again, and of their scientific predictions having failed, yet again. Now we know that the costs in terms of both public health and economic impact could well have been considerably lower if governments had acted earlier. But we can hardly say that inaction on the part of many European governments in mid-end February and beginning of March was due to lack of attention to "the evidence."

If we consider only the off-the shelf evidence, the failure to take action early on was justified, and perfectly rational. On the other hand, if we take a more expansive view of evidence, including pragmatic evidence based on judgments that take into account uncertainties and consequences, then a different picture emerges. Did the available, off-the-shelf evidence provide a justification for not taking early, precautionary measures that would have protected public health and the economy, but also would have entailed political risk? Is the ideal of evidence-based decision-making thus a cover that allowed decision-makers to weight their own political risk above the possible societal consequences of inaction?

Now we know that the costs in terms of both public health and economic impact could well have been considerably lower if governments had acted earlier.

As we have stated, one of the arguments in defense of government inaction was the lack of previous knowledge about the virus's characteristics (such as its capacity for contagion). This lack of knowledge, however, is an element that, crucially, can justify not only inaction but also precautionary action. Lack of information under certain circumstances can be a crucial argument for action. And it is public decision-makers who have to decide how to act faced with knowledge or absence of knowledge.

The relationship between evidence and decisions is complex. What counts as relevant and pertinent evidence, and how much evidence is necessary, does not have a predetermined answer, nor can it be resolved on the basis of off-the-shelf evidentiary considerations alone. Pragmatic considerations can never be left out of the equation. In fact, making explicit the inherent uncertainty in the interaction between pragmatic and off-the-shelf considerations in evidence-based decisions is one of the important lessons to be learned from the failures in managing this pandemic.

The call for evidence-based decisions is thus, in itself, meaningless. Indeed, as we see in the case of COVID-19, it is also highly political, and can be used to support inaction, even when precautionary action is warranted.