Look beneath the statistic, and you might find unexpected realities. Imagine courtesy of Brian Talley.
In an era where established facts are questioned – the massive evidence on climate change and the massive dearth of evidence on voter fraud in the United States, for example – it is worth bearing in mind the following: While believing falsehoods and outright lies and not believing truths can cause great harm, skepticism is indeed warranted when, to help make sense or easily communicate a complex set of facts, we boil those facts down into a single statistic. A line attributed (though not necessarily factually) to Mark Twain captures the point well: “Facts are stubborn things, but statistics are pliable.”
Several recent controversies, findings, and articles remind us of this crucial distinction between reality and the statistics that portend to portray reality:
* Skilled birth attendants and maternal mortality: One question that we should ask about statistics is whether they are the best measure of what they are meant to tell us. A recent article raised questions over the focus on the international community – and in turn, that of many national governments – on the proportion of births attended by skilled birth attendants as the central proxy for maternal mortality, with the policy priorities this entails. Make no mistake: There is no question that skilled birth attendants are (literally) vitally important. Evidence of their impact on reducing maternal mortality is rife (and see slide 37 of this powerpoint presentation [the link may ask whether you want to open or save the file] on the strong statistical relationship between higher skilled birth attendance and lower maternal mortality, looked at globally).
But while the correlation between the two remains, it is weaker for countries with the highest levels of mortality (see slides 38-39 of the same powerpoint presentation), as the life-saving potential of skilled birth attendants is undermined by the poor functioning health systems of which they are a part. Particularly in these high-mortality settings, might some of the resources being devoted to training skilled birth attendants (midwives, for example) be spent in other ways that would have a greater effect on reducing maternal mortality (leaving aside the availability in the world of resources for both; this ought not be an either/or choice that must be made)? For example, some analysts and development workers argue that because of the hesitancy of midwives to work in rural areas, an even higher priority than training new midwives should be “to assist community health workers, ensure basic antenatal and postnatal care and even improve local transport infrastructure — to ferry emergency cases,” along with training these community health workers to assist with deliveries.
Meanwhile, there is the question of just who counts as a skilled birth attendant – what is the underlying data that should be used for a given statistic. In one remarkable disparity, 2014 data from Sudan indicated that 78% of births were attended by a skilled birth attendant. Or perhaps only 28% of births were. The two figures were offered by WHO and UNICEF, respectively, with only WHO counting village midwives.
* Greenhouse gas emissions: Statistics cannot be relied upon if they are based on flawed underlying data. A BBC report calls into question greenhouse gas emissions statistics of several countries. Swiss scientists have detected 60-80 tons of hydrofluorocarbons, a short-lived but extremely potent greenhouse gas, being emitted from a region in northern Italy, yet Italy’s official inventory on greenhouse gas emissions, as submitted to the United Nations under the UN Framework Convention on Climate Change (article 4(1)(a)), reports as little as 2-3 tons. And China does not report any emissions of another greenhouse gas, carbon tetrachloride, because the gas is banned in China. Yet banning it does not make it disappear; despite its absence from China’s greenhouse gas inventory, 10,000-20,000 tons of the gas come out of China each year.
While these appear to be cases of failing to provide available data (manipulating data, you might say), even the best attempts to accurately capture the facts in the statistics may be thwarted by genuine uncertainties. Methane emissions from livestock, for example, are difficult to estimate. Thus, India’s listing of its livestock-related methane emissions are subject to 50% uncertainty, as the government itself notes. Nitrous oxide has an even higher level of uncertainty.
* Poverty: Uncertainty also plagues statistics on the number of people who live in extreme poverty, currently defined as earning less than $1.90 per day. In part, this is because the latest figures may be based on projections using older data. The most recent estimate of 767 million people living in extreme poverty, from last year, uses 2013 figures. Yet these figures are themselves based on household surveys in fewer than 40 countries. In other countries, figures rely on projections based on earlier data. In Africa, the continent with the highest level of poverty, these surveys were conducted, on average, only once every 6.8 years (p. 4) during the period from 1990 through 2012.
Even that data may be difficult to interpret. The world’s poorest people rarely have a regular wage. Instead of directly measuring income, surveys may measure consumption (see discussion beginning on page 4). This is itself an imprecise metric of income. Changing the questions, even to get at the same facts, can dramatically affect the poverty rate reported. When an experiment in El Salvador tweaked a set of questions to ask how much people spent on specific foods, such as plantains, mangoes, and green chilies, rather than on the more general categories of fruit, vegetables, and legumes, people’s answers changed – to the extent that the measured poverty rate fell by more than 30%.
And even when accurate, welcome statistics may cover more troubling realities. The $1.90 threshold for extreme poverty is based on the poverty lines of the 15 poorest countries. This level of income still does not get a person very much at all. As the Economist observes, “In Zambia… a person on the poverty line can afford a daily diet of two-three plates of nshima (a maize staple known as mealie meal), a sweet potato, a few spoonfuls of oil, a couple of teaspoons of sugar, a handful of peanuts and twice a week, a banana or mango and a small serving of meat,” leaving only “28% of his budget left over for other things.” Someone may be at or above the poverty line, but that is not the same as saying that that are not poor. It has been said that the poverty line might more accurately be called the starvation line.
* Official Development Assistance: Akin to the earlier question of who counts as a skilled birth attendant is what counts as Official Development Assistance (ODA). ODA increased significantly from 2012 until 2016, by some $28 billion. Yet since a 1988 rule, ODA has included “official sector expenditures for the sustenance of refugees in donor countries during the first twelve months of their stay.” Yet such spending may seem more like a social welfare program than supporting a lower-income country’s development. In an era of soaring numbers of refugees, including the number who have reached Europe, this in-country spending on refugees has become a major part of many countries’ ODA, having increased from $3.9 billion in 2012 to $15.4 billion in 2016, thus accounting for well over one-third of the overall ODA increase over the past four years. Only three of the six countries that reported reaching the 0.7% GDP target for their ODA – Luxembourg, Norway, and Sweden – would have met the target without counting in-country refugee assistance.
While the designation of this support for refuges as ODA may be misleading, the $15 billion supporting refugees in their present homes also reflects the much more positive reality that some countries are taking in and supporting refugees. In this sense, we might prefer to see even far more than $15 billion of “ODA” spent on refugees. It would just be best if the distinction was clear, and this funding was in addition to, rather than at least partially in place of, genuine ODA.
In early 2016, seven countries reported that they were using their ODA budgets for in-country refugee costs while another thirteen countries reported that while using other budgets, this support might, nonetheless, affect their ODA budgets. Might at least some countries have provided more ODA, as traditionally understood, if in-country refugee assistance did not qualify as ODA?
******************
We need statistics, but these instances remind us that we also need to be aware of their limitations, and to be attentive to the underlying facts and truths that underlie them. And they remind us, too, that we need to avoid overly sanctifying statistics to the point where our central goal is to change those statistics – increase the number of skilled birth attendants, or reduce the number of people counted as living below $1.90 per day, for example – important as both of these goals are — rather than maternal mortality, people experiencing poverty (even if they are above the poverty line), or whatever the realities that we are trying to capture may be.
