How Many Findings of the IPCC AR4 WG I are Incorrect? Answer: 28%

I suspect that headline will raise some eyebrows.

In a paper just out in Climatic Change today Rachael Jonassen and I perform a quantitative analysis of all 2,744 findings found in the three 2007 assessment reports of the Intergovernmental Panel on Climate Change.  Here is the abstract of our paper:

Jonassen, R. and R. Pielke, Jr., 2011. Improving conveyance of uncertainties in the findings of the IPCC, Climatic Change, 9 August, 0165-0009:1-9, http://dx.doi.org/10.1007/s10584-011-0185-7.

Abstract Authors of the Intergovernmental Panel on Climate Change (IPCC) Fourth Assessment Report (AR4) received guidance on reporting understanding, certainty and/or confidence in findings using a common language, to better communicate with decision makers. However, a review of the IPCC conducted by the InterAcademy Council (2010) found that “the guidance was not consistently followed in AR4, leading to unnecessary errors . . . the guidance was often applied to statements that are so vague they cannot be falsified. In these cases the impression was often left, quite incorrectly, that a substantive finding was being presented.” Our comprehensive and quantitative analysis of findings and associated uncertainty in the AR4 supports the IAC findings and suggests opportunities for improvement in future assessments.

The paper characterizes the various findings of the report in terms of the uncertainty guidance used by the IPCC.  The paper includes various summary statistics and discussion.

The answer to the provocative title of this post is found in the following part of the paper:

If we confine our attention to those findings that refer to the future, one can ask how many IPCC findings can be expected to become verified ultimately as being accurate? For example, if we consider findings that refer to future events with likelihood in the ‘likely’ class (i.e., >66% likelihood) then if these judgments are well calibrated then it would be appropriate to conclude that as many as a third can be expected to not occur. More generally, of the 360 findings reported in the full text of WG1 across all likelihood categories and presented with associated measures of likelihood (i.e., those summarized in Table 2 below), then based on the judgments of likelihood associated with each statement we should logically expect that about 100 of these findings (~28%) will at some point be overturned. 

A footnote to this paragraph explains: "This calculation assumes that each finding can be treated independently. If the findings are not independent (e.g., they are cumulative) then this calculation would result in a higher estimate."  Since this is just mathematics following from the IPCC uncertainty guidance, it should be obvious, but appears never to have been actually calculated.

What does it mean?  Nothing too interesting, really -- science evolves and any assessment is a snapshot of knowledge in time. However, I suspect that some people will get excited or defensive to learn that by the IPCC's own logic, the report's future-looking findings could include 28% or more that will not stand the test of time. Of course, such excitement and defense are part of the context that the IPCC and its critics have together created, which has led to incentives to hold the IPCC up as some sort of sacred text or to denigrate it as a sham. Our work suggests neither. Instead, from the perspective of its assessment products it is a valuable if imperfect organization.

Our paper concludes:

Although the IPCC has made enormous contributions and set an important example for global assessment of a vexing problem of immense ramifications, there remain clear opportunities for improvement in documenting findings and specifying uncertainties. We recommend more care in the definition and determination of uncertainty, more clarity in identifying and presenting findings and a more systematic approach in the entire process, especially from assessment to assessment. We also suggest an independent, dedicated group to monitor the process, evaluate findings as they are presented and track their fate. This would include tracking the relationship of findings and attendant uncertainties that pass up the hierarchy of documents within AR5. Strict rules for expressing uncertainty in findings that are derived from (possibly multiple) other findings are needed (see, e.g., the second example in the Supplementary Material).

It is not the purpose of this note to discuss other, related scientific assessments of climate change knowledge; but, we do note that our preliminary analysis of the U.S. Global Change Research Program Synthesis and Assessment Products suggests a far less systematic application of the guidance supplied to authors of those documents and far less consistent application of the defined terms. We believe that the concerns we have expressed here, and the resulting recommendations, apply more broadly than the IPCC process.

You can find the full text here.  Our full dataset of IPCC findings is online here.  Comments welcomed!

Posted in: