Objectivity in Risk Assessment:
The National Intelligence Estimate on Iran, Part 3
6 Dec 2007 in Regulatory Policy, Information Quality
The web is chock full of commentary on the recently released summary of the National Intelligence Estimate on Iran. (We emphasize summary because the body of the NIE remains classified.) We've posted here and here on the NIE as a risk assessment document, noting that it claims to be an objective assessment not confounded by risk management (i.e., policy or political) concerns.
We've read much (but by no means all) of this news and commnetary and drawn some inferences we hope are useful.
Most of the news and commentary since the NIE summary was released has been more political than analytical. Leaving aside that which is interesting only because it is scurrilous, venal and slanderous, all the accounts we have read take either a Skeptical or Credulous view toward the 2007 NIE:
- SKEPTICAL: "Iran poses a large threat, so the judgments reported in the 2007 NIE summary should be interpreted skeptically, its uncertainties highlighted, and its objectivity questioned."
- "The NIE should not be believed because the Intelligence Community, or significant elements within it, are subject to the political interference of those prefer less risk averse risk management decisions, and they are biasing the assessment of risk as a means to influence risk management policy."
- "The NIE should not be believed because judgments made by the Intelligence Community historically have been less reliable than the IC itself believes them to be."
- "The consequences of underestimating the Iranian nuclear weapons risk are more severe than the consequences of overestimating it."
- "The consequences of overconfidence are more severe than the consequences of underconfidence."
- CREDULOUS: "Iran poses a small threat, so the judgments reported in the 2007 NIE summary should be interpreted credulously, its uncertainties downplayed, and its objectivity assumed."
We take no position on which (if either) view is substantively correct. At a minimum, access to the classified portions of the NIE is essential to make that determination. Nonetheless, we note that in both the Skeptical (I) and Credulous (II) view, risk management policy conclusions are both external to and precede the risk assessment. These conclusions are unchanged by risk assessment.
- "The NIE should be believed because the Intelligence Community, or significant elements within it, are less subject to the political interference of those who prefer more risk averse risk management decisions, who previously biased the assessment of risk as a means to justify risk management policies unsupported by unbiased risk assessment."
- "The NIE should be believed because the Intelligence Community has fixed previous problems that led to unreliable risk assessment."
- "The consequences of underestimating the Iranian nuclear weapons risk are less severe than the consequences of overestimating it."
- "The consequences of overconfidence are less severe than the consequences of underconfidence."
Although these perspectives might seem to be irreconcilable, they share at least the following important (and disturbing) characteristics:
- "The purpose of risk assessment is not to assess risk, but to justify a predetermined risk management policy, prevent the establishment or implementation of an opposing risk management policy, or to reverse such a policy already in place."
- "Uncertainty in risk assessment matters only with respect to how it affects opponents' ability to establish and implement their risk management policies."
- "Professions of objectivity are embraced if risk assessment supports a preferred risk management policy and contested if it supports an opposing risk management policy."
Miles' Law states: "Where you stand depends on where you sit." According to Rufus E. Miles, he coined the term while working at the Bureau of the Budget (the predecessor to today's Office of Management and Budget) in 1948 or 1949. A subordinate of Miles had been offered a raise if he left BOB to work for the agency he then was supervising, and he sought to use that offer as leverage to obtain a raise at BOB and stay put, which he strongly preferred to do because a position at BOB was much more prestigious. Miles refused, and the subordinate chose to trade prestige for money.
While at BOB Miles' subordinate had been severely critical of the agency. After he left, Miles predicted -- correctly -- that his erstwhile subordinate would completely change his view:
A day or two after he departed, I said to one of my associates, "Just watch! Within three or four months he will be as critical of the Bureau of the Budget and as defensive of his agency as he has been the opposite within the Bureau." My associate was astonished at me. "Oh no!" he said, "he is much too objective and fair minded for that kind of turnabout." My rejoinder was, "You should not dispute whether this will happen, but only how long it will take." It took about two months longer than I estimated, as I recall it. When it did happen, I said to my associate, "You see, it depends on where you sit, how you stand."
Miles' Law certainly applies to those who were Skeptical in 2005 but Credulous in 2007, and vice versa. It's likely that it also applies to many of the Intelligence Community analysts (and their supervisors) who prepared these reports. The challenge for risk assessors -- whether they are dealing with part per billion exposures to tropospheric ozone, one in a million airport runway incursions, or the Iranian nuclear weapons program -- is to become objective in the assessment of risk and remain so irrespective of competing risk management views, the proponents of which all seek our blessing on their conclusions.
Miles, RE, Jr. The Origin and Meaning of Miles' Law," Public Administration Review, Vol. 38, No. 5 (Sep. - Oct., 1978), pp. 399-403.
