On October 28, 2009, the Office of Management and Budget solicited comments on its implementation of the Paperwork Reduction Act. The purpose of the PRA is to minimize burdens on the public resulting from the federal government’s information requests.
Neutral Source managing editor Richard Belzer submitted comments on his own behalf. These comments eventually will be uploaded by OMB to Regulations.Gov, the Federal government’s web portal for all regulatory matters. (Clicking on the link above will reveal a fundamental weakness of the web portal: Unless the agency chooses to include information identifying the name and organizational affiliation of the submitter, there is no way to find any specific comment without opening them all.)
In response to numerous requests, the text of this comment is reproduced below in four parts plus references and footnotes.
Richard B. Belzer, Ph.D.
PO Box 319
Mount Vernon, VA 22121
December 28, 2009
The Honorable Cass R. Sunstein
Office of Information and Regulatory Affairs
Office of Management and Budget
Washington, DC 20503
Dear Administrator Sunstein:
Thank you for providing the public an opportunity to comment on the federal government’s performance implementing the Paperwork Reduction Act (PRA), and soliciting ideas concerning how it can be improved. Your request has been a long time coming: To the best of my knowledge and experience, which consists of ten years serving as a civil service economist in the Office of Information and Regulatory Affairs (OIRA) and another ten years outside the government, there has never been a previous such request outside of rulemaking.
Many of the problems with current PRA implementation result from defects in transparency. OMB culture and practice encourage and reward discretion and confidentiality, and as a result, inevitably discourage transparency. This is exacerbated by OIRA’s executive role in reviewing draft proposed and final regulations¾a task that must remain privileged and confidential to be effective and deserve the trust of the President, OIRA’s exclusive client when it acts in accordance with this role.
There are important negative externalities associated with executive regulatory review, however, and these externalities undermine OIRA’s effectiveness with respect to implementing its statutory responsibilities under the PRA. Too often, these statutory responsibilities have been subordinated to the demands of regulatory oversight. In my comments below, which reiterate some of the suggestions I made in previous comments to OMB concerning centralized regulatory review, I offer suggestions that would improve OIRA’s ability to implement the PRA, significantly improve the quality of information obtained by agencies in accordance with the law, and enhance regulatory review at the same time.
OIRA seeks comments on “how to improve the current situation,” with special emphasis on several matters related to estimating and reporting burden, reducing the magnitude of burden, “maximizing the utility of information,” and “improv[ing] the PRA review process.” These requests are rather vague, and if there are specific issues that led to this request, OIRA has been characteristically elliptical in its descriptions. For that reason, I am interpreting the request for comment broadly.
I provide these comments on my own behalf. They do not necessarily represent the views of Regulatory Checkbook, a nonprofit organization that I serve as president, nor do they reflect the views of or were prepared at the request of any third party. Where my comments concern specific Information Collection Requests, they arise because of my own experience.
I. SOLUTIONS IN SEARCH OF PROBLEMS
Much of the literature in this area consists of recommendations for changes in law, regulation, or administrative practice that have as a common feature their instrumental character. That is, these recommendations appear to be motivated for purposes other than improving the administration of the Paperwork Reduction Act, which include minimizing paperwork burdens on the public; ensuring the greatest possible public benefit from and maximizing the utility of this information; and strengthening decision-making, accountability, and openness in Government and society. In many cases, recommendations are self-serving (such as to exempt one’s favored constituency from having to comply), contrary to law (such as eliminating OIRA’s interest in and focus on information quality), or founded on systemic antipathy to OIRA. Before examining bona fide problems with agency compliance and OIRA implementation of the PRA, it is useful to mention these instrumental recommendations for the purpose of dismissing them.
A. Weaken or abandon OIRA review procedures for selected (?) voluntary ICRs.
In a recent monograph it was suggested that the quality of Regulatory Impact Analyses could be improved if certain voluntary ICRs were exempted from OIRA review. Perhaps coincidentally, the ICRs that the authors want to exclude happen to be ones in which OIRA enforces the standards of the PRA on their own government-funded research – in particular, willingness-to-pay (WTP) surveys using contingent-value and other stated-preference methods for estimating the value of public goods:
Many researchers who do federally supported research on the benefits and costs of regulations, as well as federal agency personnel who are responsible for developing and supporting economically justified regulations, report horror stories about extensive delays in getting surveys approved. Virtually all would agree that the required public comment on surveys is an unwarranted and unwelcome intrusion on research autonomy. Most would further agree that the PRA and OMB’s interpretation of its requirements are a little too energetic and could be tweaked to make data collection to support regulation more efficient without compromising the goals of the PRA.
The authors provide no documentation to support the alleged “horror stories,” do not identify any of the “many” researchers who experienced them, and do not even hint as to why they might have occurred. Nor is there any evidence that the authors consulted with OIRA staff to ensure that their understanding of the facts was accurate.
By law, government-sponsored research¾including WTP surveys¾ must meet minimum standards for practical utility, statistical rigor, and information quality irrespective of whether it is conducted by agency personnel or by researchers employed by universities or think tanks. Nongovernmental researchers need not endure any “intrusion” on their “autonomy” if they secure private funding for their work. Taxpayer research funding is not an entitlement, and it creates certain additional responsibilities to maximize quality.
In short, this “reform” proposal is an undisguised plea for special treatment. In particular, some economists who perform government-sponsored WTP studies dislike having their research designs reviewed by OIRA. Nearly all researchers who perform government-sponsored surveys would like to get out from under the review process, too.
B. Weaken or abandon OIRA’s high statistical quality standards.
OIRA’s “energetic” defense of longstanding PRA standards in the case of WTP studies did not occur in a vacuum. It occurred because of documented instances in which the results of government-funded WTP surveys approved for a relatively undemanding information quality purpose (e.g., methodological research) were subsequently used improperly by an agency for a very demanding purpose (e.g., estimating the benefits of air pollution regulations). An example with which I am personally familiar is a methodological study designed to estimate WTP using risk-risk instead of risk-dollar tradeoffs. The authors used a convenience sample that was entirely reasonable for a methodological study, but which is utterly inappropriate for benefits estimation. Indeed, the authors explicitly warned against using their results for estimating regulatory benefits:
While the results of the application of our new morbidity valuation methodology to chronic bronchitis are encouraging, much further research is needed before applying the methodology to give estimates precise enough to be used in regulatory analyses.
The Environmental Protection Agency did not heed this advice. Instead, the Agency has repeatedly used estimates from this study as the basis for valuing the prevention of chronic bronchitis via air pollution control regulations. Moreover, the EPA is not alone in having misused these estimates. Harrington et al. do so as well. They cite this methodological study as “[t]he best available estimate of WTP to avoid a case of chronic bronchitis.”
The underlying problem these economists are complaining about is OIRA’s insistence on randomized sample designs sufficient to extrapolate results to an identified population. The example above vividly illustrates why OIRA must be so demanding. If OIRA approves a convenience sample to study “unsettled methodological controversies,” current procedures do not provide any way to prevent the results of convenience samples from being misinterpreted as representative. Not only will some agencies happily do this if sufficiently inclined, the public cannot count on knowledgeable academic researchers to object.
Other researchers have been able to satisfy these high standards even though their surveys are much more burdensome, and in some cases, personally invasive. It is not more difficult to develop representative samples for benefits valuation research than for gathering economic statistics or performing exposure epidemiology. WTP researchers should not be asking for special treatment to make up for inherent defects in their preferred research designs.
Given this record, it is appropriate for OIRA to review prospective contingent valuation studies with the expectation that the results will be used improperly. It is unfortunate that responsible researchers suffer negative externalities associated with others’ abuse of their work. That Harrington et al. do not seem to be interested in this problem, and are content to misuse results themselves, is sufficient ground for rejecting their recommendation.
Even though OIRA’s insistence on high statistical standards is justified, there may yet be a way for OIRA to relax its statistical standards in cases where sponsored research is strictly methodological and never is used for broader purposes such as estimating regulatory benefits. The challenge is to devise enforcement mechanisms that will effectively and inexpensively prevent the misuse of such data. Any mechanism that depends on OIRA always playing the role of “bad cop” is destined to fail, for despite its reputation for saying “No,” OIRA is institutionally not eager to do so and usually finds it politically unwise.
A workable compromise has several required elements, the violation of any one of which rendering the deal unworkable. First, OIRA staff must diligently recite in Terms of Clearance the sampling methodology that they are approving and list all material restrictions that this design places on subsequent use of data. This is necessary to create a permanent record documenting the conditions of approval. Second, agencies must prominently disclose these Terms of Clearance every time they discuss the ICR or disseminate results. If they fail to do so, or the agency’s portrayal is inconsistent with them, dissemination must be deemed a per se violation of applicable information quality guidelines subject to immediate rescission upon the filing of a petition seeking the correction of noncompliant information. Third, OIRA must enforce these Terms of Clearance when it reviews Regulatory Impact Analyses and other factual or analytical material it receives in support of draft regulatory actions, and ensure full compliance with the Information Quality Act. Fourth, agencies must maintain permanent URLs to Terms of Clearance and require that every publication include prominent links to them in the acknowledgements section.
In many cases, strict compliance with every element of the PRA, the Information Collection Rule, and any applicable terms of clearance isn’t necessary or desirable. But the absence of full compliance should shift the burden of proof to the agency to document and defend, prior to dissemination, every departure from what OIRA approved with clear and convincing evidence of substantive harmlessness. If an agency fails to meet this burden of proof, or worse, fails to even make a credible case, then the use of the information should be subject to the strictest possible scrutiny.
C. Weaken or abandon requirements for public comment on selected ICRs.
Harington et al. (2009) also seek lax standards for WTP surveys with respect to public comment. They justify special treatment on the ground that these ICRs are “quite technical in nature” and thus are beyond the comprehension of “self-selected laypersons” who “rarely have much useful to add.” They further complain that the public comment process invites participation by interest groups, failing to recognize that interest groups often have more expertise than the general public and are generally better equipped to predict when and how results could be misused.
The claim that WTP surveys are so technical that they should be excluded from public comment is risible. Among the ICRs I have personally reviewed, those dealing with WTP surveys are among the least technical. Moreover, WTP surveys I reviewed during my tenure at OIRA tended to have very low quality, relying on survey designs incapable of passing scope and additivity tests. They had little or no practical utility and deserved to be disapproved.
In any case, it is contrary to the purposes of the Paperwork Reduction Act to give special treatment to elites just because they pronounce themselves experts. Nothing in the law or OMB’s Information Collection Rule prevents bona fide subject matter experts from submitting comments during either the 60- or 30-day review periods. In my OIRA tenure, I welcomed such input and often sought out experts myself instead of waiting for them to send public comment letters. The most useful expert comments I received came from genuinely independent sources, however, not testimonials spun up by the researchers whose surveys are under review.
Sponsoring agencies and OIRA staff alike can readily distinguish between public comments that are focused on well-defined PRA issues and those that are not. The PRA gives OIRA certain carefully circumscribed authorities that do not include the consideration of policy issues that interest groups often raise. Thus, OIRA has little difficulty disregarding public comments that ask it to take actions beyond its authority. Furthermore, it may be true that interest group comments dealing with non-PRA issues add little value at the margin, but this is actually an argument for OIRA to do a better job educating the public about how to participate in the PRA process more effectively. As I point out in Section III(B)(2) below, OIRA’s educational efforts since 1980 have been less than negligible.
Finally, if OIRA were to exempt WTP surveys from public comment, it would validate the hegemony of a cabal of government–funded contingent valuation researchers. It would open the floodgates to endless similar claims from other researchers seeking special treatment. Significant controversy has arisen recently concerning the extent to which scientists in certain controversial fields might have succeeded in excluding intellectual opponents from publication in peer reviewed journals and other forums of scholarly debate. It is critical that OIRA avoid taking any action that could suggest it is a party to such exclusionary conduct.
D. Replace public comment with peer review.
If OIRA cannot bring itself to exclude the public from ICR review, Harington et al. (2009) call on it to replace public comment with peer review. This suggestion also is ill advised, for additional reasons beyond those mentioned earlier. Public comment and peer review serve fundamentally different purposes. Whereas peer review, properly applied, may ensure that researchers can credibly answer the questions they plan to address, public comment is the proper venue for asking, among other things, whether the questions themselves are worth addressing.
Not only are these processes different, they are not in conflict. Nothing in either the law or OMB’s Information Collection Rule prevents agencies from arranging for rigorous, independent, and external peer review at any step in the ICR process. It is not OMB’s responsibility to arrange for peer review during its 60-day review period if the sponsoring agency has decided against it. Indeed, when it encounters submissions that ought to have been peer reviewed prior to submission, OIRA should seriously consider summary disapproval. The PRA review process is not made more efficient or effective when agencies submit ICRs in haste.
E. Replace OIRA statisticians with nongovernmental peer reviewers.
Since 1980 it has been OIRA’s statutory responsibility to disapprove information collections that lack practical utility. Harrington et al. imply that OIRA exercises this authority too aggressively, but the only evidence mustered consists of anecdotal “horror stories” that they decline to document. It is OIRA’s executive responsibility to ensure that regulatory impact analyses rely on high quality statistical information. The OIRA staff is correct to complain about the poor quality of information in agency Regulatory Impact Analyses, much of which was obtained outside the PRA process, and disapprove ICRs that would only increase the quantity of poor quality information.
As with their previous recommendations, this one is transparently self-serving. There is no obvious reason why peer review by nongovernmental experts in lieu of OIRA review would result in WTP surveys that have greater practical utility. It is a rare academic peer reviewer who has even heard of practical utility, the statutory standard that OIRA is charged with implementing. In my experience, academic researchers hardly ever are willing to stand in the way of low quality research unless it reduces funds available to finance their own work.
We can predict what would happen if this change were adopted. Researchers A, B, and C would have this year’s crop of surveys “peer reviewed” by Researchers D, E, and F. Next year, Researchers D, E, and F would have their surveys “peer reviewed” by Researchers A, B, and C. Everyone’s surveys would be approved, and practical utility would decline.
F. Permit cash incentives for survey participation.
Harrington et al. (2009) also object to OIRA’s unwillingness to allow cash incentives to encourage participation, and on this point they reside on solid ground. Nothing in the PRA or OMB’s Information Collection Rule justifies this practice. OIRA’s resistance is driven by OMB’s institutional interest in restricting federal spending, not a genuine concern that cash payments would lead to bias.
In the mid-1990s, I shepherded EPA’s National Human Exposure Assessment Survey (NHEXAS) ICR through the OIRA review process and encountered this very problem. By its very nature, NHEXAS was predestined to be highly burdensome. It involved the collection of enormous amounts of information on personal exposures to environmental contaminants. Portions of the information collection were necessarily invasive, such as the collection of blood. Yet the project could not be successful without a very high response rate, and OIRA statisticians opposed researchers’ plans to incentivize participation with cash payments. The statisticians’ objections were clearly nonscientific and nonstatistical. It was not credible to believe that research subjects would, for example, alter their blood lead levels to gain compensation.
The Region V NHEXAS study offered $5 for completion of a baseline questionnaire, plus an additional $15 and $40, respectively, for completion of “core” and aerosol monitoring. Raffle tickets also were issued to create an additional implicit cash incentive. These incentives surely helped, but they were only barely sufficient. Federal statistical policy calls for studies that fail to achieve an 80% or better response rate conduct a thorough analysis of nonresponse bias. To their credit, NHEXAS researchers did this. Had they been allowed to give nontrivial cash payments, however, they should have been able to easily exceed OIRA’s performance standard.
Researchers performing WTP surveys appear to have been less successful in securing high response rates despite their obvious advantages compared with invasive studies like NHEXAS. There are two likely reasons. First, WTP surveys lack enough salience to motivate high participation. Second, achieving high response rates has not been a priority among WTP researchers.
G. Amend the law.
Most changes recommended by Harrington et al. (2009) would require congressional action. Others also have recommended statutory changes. For example, in one of several comments to OMB on its request for comment on regulatory review, a large group convened by OMB Watch said the PRA needed to be “amended and reauthorized.” Which provisions they think ought to be amended, however, they did not make clear.
It is inappropriate to amend the PRA based on anecdotes, and premature to consider it when existing law and implementing regulations are not enforced. Even opening up such a discussion at this point is dangerous folly.
H. Burden-hour budgeting.
Bass et al. (2008) also propose that OMB “set an annual burden-hour budget that would allow the agency flexibility to collect information on issues as it sees fit without OIRA’s approval as long as it is within the budget” (p. 27). This proposal is somewhat analogous to one often made by former OIRA Administrator and OMB Director James C. Miller III, who has long advocated the adoption by Congress of a budget for regulatory expenditures.
The Bass et al. proposal is certainly innovative but it appears to have all the implementation problems of a regulatory budget, plus others. For example, Miller would preserve OIRA’s role in monitoring agency cost estimates, in hopes of preventing them from being systematically underestimated by the agencies. Whether OIRA could do this is a matter of conjecture, but its limited success in removing bias from cost and benefit estimates in RIAs does not inspire confidence.
In contrast, Bass et al. appear to intend that ICRs under the budget cap would be exempt from OIRA review. As hard as it would be for OIRA to prevent biased estimation of regulatory costs in a regulatory budget regime, removing OIRA from the review of agency paperwork burden estimates would ensure that the downward bias now observed by OMB would get much worse. Agencies already have weak incentives to estimate burden accurately, in large part because OIRA review often fails to detect bias or enforce corrections when errors are found. Without any OIRA oversight, however, a burden-hour budget is predestined to fail because it would be a budget without limits.
I. Online crosswalk of ICRs with regulations.
Bass et al. (2008) offer another suggestion that seems much more promising. They propose that OMB fix the current regulatory tracking system to include an online, searchable, and well-maintained database of all regulatory actions in the pipeline, including references to all related ICRs (p. 50). Currently, the only public inventory of ICRs is the one maintained by OIRA on Reginfo.Gov, a clunky system that is cumbersome to use and provides the barest of search utilities. At best, skilled users can find what they want only if they know what they are looking for. Just to locate an ICR, one must know in advance the sponsoring agency and either the ICR title or its OMB Control Number. Information collections may or may not be packaged into logical packages within a single Control Number. From there, it is a grueling task to discover the statutory authority and regulatory action (if any) on which it the ICR is based. This database was designed by and for OIRA staff and other insiders, not for the general public, and has limited practical utility befitting the fact that it reflects the state-of-the-art¾circa 1998.
II. Examples from Personal Experience
My comments in this Section are focused on personal experiences serving in OIRA and in various capacities after my departure in 1998. These experiences undoubtedly reflect a selection bias insofar as the paperwork issues I have worked are not drawn randomly from the population of all ICRs reviewed by OIRA. Nonetheless, my selection bias is not that different from other commenters except that few others have worked on paperwork issues both outside the government and within OIRA. Moreover, my focus is on examples of deficient practice because that is what OIRA’s request for comment has sought from the public.
A. U.S. Patent and Trademark Office ICRs
I recently conducted an extensive review of the paperwork burdens of a series of regulations put forward by the U.S. Patent and Trademark Office (USPTO). Three proposed rules were represented by USPTO as “not significant,” and thus exempt from OMB review under Executive Order 12,866. Two proposed rules were classified as “other significant,” meaning that neither was likely to have effects greater than $100 million in any one year. They were thus exempt from the Regulatory Impact Analysis requirement Executive Order 12,866 imposes on economically significant draft regulations. USPTO promulgated these as a single final rule, once again with an unsupported claim that the rule was not economically significant.
Each of these Executive Order 12,866 certifications was false. Each rule would have forced fundamental and costly changes on the U.S. patent system. More to the point of this comment letter, the false denial of economically significant costs led USPTO to make false Paperwork Reduction Act certifications in the relevant 60-day notice. USPTO knew that each rule would have created massive new paperwork burdens because it designed each rule for this express purpose. Thus, it is highly unlikely that any of USPTO’s false claims about paperwork burdens was accidental.
Two of the three “nonsignificant” proposed rules included a claim that no new paperwork burdens would be created, and thus no revision to an existing ICR needed to be submitted to OMB. I estimated that annual paperwork burdens for the proposed versions of these rules ranged from a low of $820 million per year to a high of $4.4 billion per year. Perhaps more disturbingly, I discovered that USPTO lacked a valid OMB Control Number for the baseline paperwork requirements. In short, the Patent Office has been imposing hundreds of millions of dollars in annual paperwork burdens without OMB approval roughly since the PRA was enacted in 1980.
Despite USPTO’s successful evasion of OIRA and public accountability through the use of false certifications and misleading notices, engaged public commenters identified and provided credible preliminary estimates of the paperwork burdens that the Patent Office had said did not exist. The Patent Office responded to these comments not by admitting error, but largely by ignoring or dismissing the commenters.
To its credit, OIRA declined to issue an approval of the information collections contained in one of these ICRs. This decision was almost certainly based on the demonstration by commenters (including me) of overwhelming evidence of USPTO’s systematic procedural and substantive noncompliance with the PRA. Inexplicably, OIRA did not perform the simplest task most consistent with its statutory authority: disapprove the ICR and direct the Patent Office to start over. Instead, OIRA issued an OMB Control Number approving, without further public notice and opportunity to comment, baseline information collection elements that we commenters had flagged as illegal and for which USPTO had never produced objectively supported burden estimates and accepted a revised Supporting Statement that, like its predecessors, responds to public comments with risible disingenuousness.
There is evidence that USPTO is slowly reforming its conduct. The Patent Office recently published an advanced notice of proposed rulemaking seeking comment on “possible revisions” to the administratively stayed final rule that led to these PRA violations. Alas, the notice contains the usual boilerplate PRA statement, but no information on which the public can comment, and no acknowledgement of Executive Order 12,866. These revisions do not change the virtual certainty that the regulation would have astoundingly burdensome paperwork and be economically significant, this requiring a Regulatory Impact Analysis.
OIRA should not be covering up for agency misfeasance, malfeasance, and nonfeasance. When it does, OIRA undermines confidence in its ability and willingness to fairly and effectively administer the Paperwork Reduction Act and thereby discourages the public from taking the Act seriously. When it behaves in ways contrary to the letter and spirit of the law, OIRA abuses its statutory authority to manage the paperwork process without judicial review and gives credence to the arguments of those who do not care about minimizing paperwork burdens on the public or maximizing the quality of information the government collects.
2007-2009 Rule Makings in Which USPTO
Falsely Denied Economic Costs and Paperwork Burdens
|Notes: aSubmitted as draft final to OMB and cleared on December 10, 2007.bPromulgated as a single final rule. cInterim Regulatory Flexibility Act Analysis (June 2007).dSubmitted as draft final to OMB and cleared on September 9, 2008.
e No valid OMB Control Number for pre-existing information collection (i.e., pre-existing collection was a “bootleg”).
B. Environmental Protection Agency ICRs
Although I have reviewed numerous EPA information collections, three stand out as worthy of special mention.
1. The National Human Exposure Assessment Survey (NHEXAS)
When it was submitted to OMB in the early 1990s, NHEXAS was the most ambitious effort ever mounted to obtain valid and reliable data on human exposure to environmental agents. Prior to NHEXAS, EPA’s practice was to rely on default exposure assumptions for risk assessment, a practice that has been widely criticized, including by OMB.
I shepherded NHEXAS through the ICR clearance process. OMB statistical policy requires that such samples obtain an 80% response rate or include at the outset credible plans for estimating and adjusting for nonresponse bias. Some method for enhancing response rates was essential to preserve the ability to draw inferences about a population from a representative sample, but informal OMB procedures at the time strongly discouraged cash payments. As noted above, I encountered internal opposition within OIRA regarding the provision of incentives to increase participation and avoid nonresponse bias. A compromise was arranged in which researchers could offer an array of alternative (albeit small) incentives. This made sense because NHEXAS was a pilot project. Researchers were testing a number of innovative exposure assessment techniques, and this enabled them to test comparative incentives as well.
As submitted, the NHEXAS ICR had significant PRA defects. My job was to identify ways to enhance its adherence to PRA principles without compromising (and if possible, improving) its scientific merit. I accomplished this by meeting with both EPA staff and the academic researchers who would be performing the studies, and taking the initiative to contact outside experts to get advice. The result of this collaboration was a much improved project that the needs of EPA and the field researchers within the requirements of the PRA.
I believe this is the right way for OIRA to manage ICR reviews involving complex scientific and statistical information. Ideally, agencies would not postpone discussions with OIRA until submission of the ICR and OIRA staff would not wait for agencies to submit ICRs when they discern a need for better quality information. Instead, OIRA should identify data needs based on its understanding of the Regulatory Agenda and take the initiative to launch and support high quality information collection activities. Equally important, however, OIRA cannot perform its PRA function effectively if it treats ICRs as internal government matters and discourages the public from participating.
2. The Endocrine Disruptor Screening Program Tier 1 Test Order ICR
I recently supplied comments to OIRA on the 30-day notice for EPA’s Tier 1 Test Order ICR for the Endocrine Disruptor Screening Program. In these comments, I showed that EPA: failed to document that the laboratory tests to be required have been validated for their stated purpose (as required by the law authorizing the information collection); failed to demonstrate actual practical utility (required by 5 C.F.R. § 1320.5(d)(iii), referencing § 1320.3(l)); and failed to provide objectively supported burden estimates (required by 5 C.F.R. § 1320.8(a)(4)), among other defects. OIRA approved the ICR despite these violations of law and the Information Collection Rule.
OMB’s Terms of Clearance contain language that can, with effort, be construed as threading the needle to allow EPA to impose Tier 1 Test Orders only in cases where doing so does not violate the PRA. However, it will be difficult for affected parties to invoke this language in defense of their legal rights under the Paperwork Act not to be subjected to expensive demands for information that are duplicative and lack demonstrable practical utility. Nothing in OIRA’s Terms of Clearance explains how the conditions therein will be enforced.
3. Pollution Abatement Costs and Expenditures (PACE) Survey
On and off for perhaps 20 years, EPA has sponsored this survey to obtain information it believes it needs to estimate the aggregate cost of environmental protection. The U.S. Census Bureau administers the survey on EPA’s behalf. Results for the 2005 information collection were published in 2008.
The PACE survey has long suffered numerous practical utility problems, beginning with the fact that although its stated purpose is to help EPA estimate cost, cost isn’t defined in economic terms. The instructions for completing the survey run for 28 pages. The sheer complexity of the task invites respondents to fill in the blanks with figures that are arbitrary or convenient. No two respondents will use the same definitions, meaning that all aggregate “cost” estimates derived from PACE have no valid interpretation.
These fatal substantive defects aside, there is a procedural aspect of PACE that is much more disturbing. The Census Bureau tells respondents that responses are required by law:
Is your response to this survey mandatory?
Yes. Responding to the PACE survey is required by law (Title 13, United States Code, Sections 131, 182, 193, 224, and 225). You may visit our website at www.access.gpo.gov/uscode/title13/title13.html.
This is an egregious abuse of the Census Bureau’s statutory authority in violation of the Paperwork Reduction Act. It is raw intimidation. None of the statutory references applies to a survey performed by the Census Bureau on contract to another agency for purposes unrelated to the Bureau’s constitutional and statutory responsibilities.
In short, the Census Bureau has discovered that renting its statutory authority to compel participation is a profit center. One can only imagine the outcry that would result if the Internal Revenue Service were to go into the business of supplementing its appropriations by collecting money from other agencies to append unrelated information collections to its Form 1040. Unlike the Census Bureau, which if it actually penalized someone for not returning a PACE survey it would collect a mere $500, the IRA could prosecute nonrespondents for felony tax evasion.
This example could be an aberration, but no one knows. OIRA should immediately identify every agency with statutory authority to compel the public to provide information, and review each instance in which the agency invokes this authority to ensure that it is clearly within the tailored boundaries of its authority. Because of PACE, a good place to start is with the Census Bureau, to determine how much profit it currently earns by abusing the public on behalf of other agencies.
C. Epidemiological surveys
During my OIRA tenure I reviewed numerous epidemiological surveys sponsored by the Agency for Toxic Substances and Disease Registry (ATSDR). I observed a fairly consistent pattern of poor research design compounded by unrepresentative, convenience sampling. For example, many surveys used as a proxy for chemical exposure the linear distance between the boundary of a Superfund site and a person’s residence. By obtaining from respondents a laundry list of unverified health effect claims, ATSDR could discover associations between hazardous waste and disease even in the absence of exposure. In every instance in which I observed this research design, I strongly recommended that the survey be disapproved.
Since leaving OIRA, I have continued to review epidemiological surveys more broadly across the federal government. Many of the systematic design defects I observed in 1990s-vintage ATSDR epidemiology I have seen elsewhere. These surveys do not appear to garner the level of attention from OIRA staff that they should, probably because each one is small and its potential harm may seem inconsequential. This is a mistaken impression. Bad design metastasizes from small surveys to larger ones. When a precedent for low quality is established in a small survey, it becomes easier to justify low quality more generally. A peculiar version of Gresham’s Law arises: Because it can be done so much more cheaply, bad science drives out good science at the earliest stages of government-sponsored research.
III. PROBLEMS IN SEARCH OF SOLUTIONS
An essential prerequisite for solving a problem is to define it clearly and accurately. This is an integral part of OMB’s Information Collection Rule, which directs agencies to demonstrate that the information they seek to obtain from the public has practical utility, meaning “actual, not merely … theoretical or potential usefulness.” To make such a demonstration, agencies must understand the problem that they seek to solve and develop a cogent case explaining why obtaining the information they propose to collect would solve it. Similarly, careful problem definition has long been a hallmark of presidential regulatory principles. Not all well-characterized problems have solutions, but there are no solutions for problems that are not clearly or accurately defined.
Based on my experience, there are three types of generic problems with OIRA’s implementation of the PRA. The first type consists of errors committed by agencies, but which OIRA tolerates. The second type consists of problems directly attributable to OIRA, which do not have an underlying agency origin. The third problem type is the product of the first two: Agency and OMB actions disenfranchise the public by discouraging its active and effective participation.
A. Deficiencies in Agency Planning, Consultation, and Implementation
The PRA and OMB’s Information Collection Rule require agencies to engage in a number of activities as part of the discipline necessary to avoid unreasonable paperwork burdens on the public. Many problems with the PRA arise because agencies do not fulfill these obligations, and OIRA tolerates their malfeasance, misfeasance, and nonfeasance.
1. Agency head/senior official disinterest.
The Information Collection Rule requires agency heads to designate “Senior Officials” to carry out the responsibilities delegated to them by law. The purpose of requiring a direct reporting relationship is to ensure that paperwork reduction enjoys a high priority within the agency.
The senior officials nominally assigned this responsibility are chief information officers, general counsels and solicitors. However, actual responsibility for PRA compliance is delegated several layers below, often to agency staff who might not have much contact with the Senior Official and almost certainly no contact with the agency head. Unless and until the importance of the PRA is restored to its original stature within the agencies, it is unlikely that paperwork matters will regain the traction they once held. OMB should audit the agencies’ assignment of personnel to PRA planning and implementation, the degree to which they actually report or have access to their Senior Official, and whether they perform a merely administrative function that prevents them from actually fulfilling the duties required by law.
One way for OIRA to get Senior Officials to pay more attention to paperwork issues is to compel them to respond personally when Desk Officers experience compliance problems. If responding to problems becomes a significant burden, they will be more inclined to take whatever internal actions are necessary to reduce it.
2. Failure to actually perform required planning functions.
Agencies are forbidden to conduct or sponsor an information collection unless they have taken the following major planning steps:
- Review each information collection to evaluate its need, describe its functional purpose, develop a collection plan, prepare an objectively-supported estimate of burden, look for ways to reduce burden through electronic means, conduct pilot tests where appropriate, develop a plan for effective and efficient information management; ensure that it is inventoried and displays a valid OMB control number, provide proper notice and information disclosure to the public, and complies with the applicable statutory clearance process;
- Evaluate and summarize the public comments received; and
- Submit all required documentation to OMB, include a certification by the agency head (or approved designee) of compliance with all applicable procedural requirements.
Information must be provided “in a manner that is reasonably calculated to inform the public.”
This list of procedural steps might seem daunting at first, but it is important to remember that the purpose of the Paperwork Reduction Act was, and remains, to regulate Federal agencies’ otherwise unbounded proclivity to deputize the public as unpaid research assistants. Moreover, agencies have about 30 years’ experience managing these responsibilities since the PRA was enacted in 1980. By now, compliance with PRA procedures should be second nature, having been thoroughly and completely grafted into agencies’ internal policies and practices.
As part of the audit recommended in subsection 1 above, OMB should rigorously evaluate whether each agency actually performs the planning functions set forth in the Information Collection Rule. There is considerable evidence that agencies are not engaging in the law’s required planning activities as frequently as they are certifying that they comply with them.
3. “Bootleg” information collections.
A recurring problem since the Paperwork Reduction Act was enacted in 1980 is the propensity to impose paperwork burdens without first obtaining valid OMB Control Numbers. In every annual edition of the Information Collection Budget, OMB reports on its efforts over the past year to combat the illegal imposition of paperwork burdens¾what the OIRA staff historically have called “bootlegs.”
It has been an OIRA goal for many years to reduce to zero the incidence of bootlegs, but to date it has lacked an effective way to even detect them. OIRA is impeded in this effort by the near universal ignorance about the PRA within both the general public and those who are subject to information collection requests. That ignorance, in turn, is largely the product of OIRA’s own lack of educational effort, its tolerance of rote adherence by the agencies to applicable law and regulation, and its willingness to cover up for agencies caught violating the law.
Discovering bootleg information collections is not easy. Few members of the public have ever heard of the PRA, much less are capable of recognizing an illegal information collection when they see one. Fewer still care enough to complain, or even know where to send a complaint so that it will receive proper attention.
OIRA incentivizes bootlegs by discreetly covering up for the agencies that commit them. A brief notice is made of the event in the Information Collection Budget¾a document hardly anyone ever reads¾and OIRA issues a new OMB Control Number that makes the collection valid from that date forward. OIRA polices bootleg information collections like a lazy county sheriff who apologetically issues moving violations only when there are too many witnesses, and makes up for it by being sure he’s out fishing instead of appearing in court.
If OIRA is serious about incentivizing agencies not to commit bootlegs, it needs to make a greater show of publicizing and penalizing them. If OIRA is serious about detecting bootlegs, it needs to create a simple web-based utility enabling the public to provide anonymous tips¾then educate the public about the PRA and promote the use of the tip line.
4. Routinely false certifications.
Every ICR must be accompanied by a certification of compliance with applicable PRA principles and procedures. There is reason to be concerned that these certifications have become rote, mechanized, boilerplated, and thus ineffective.
Agency certifications take the form of a checkbox-style statement. These certification statements are rarely, if ever, supported by evidence. Indeed, OIRA’s procedures require agencies to provide documentation only if the agency is unable to certify compliance. No one knows how often agencies admit that they are unable to certify compliance. There are no penalties for invalid or inaccurate certifications, so there is every incentive to certify falsely. It would be unsurprising to learn that, regardless of the facts, the number of checkboxes not checked is exactly zero.
As the agencies’ paperwork “regulator,” it is OIRA’s responsibility to do what regulatory agencies do: Regulate. A case can be made that OIRA lacks sufficient staff resources to regulate. The steady erosion in the number of OIRA professional staff, which has declined by about half since the early 1908s, surely has not made the task easier. If OIRA is destined to be understaffed because of political constraints on the absolute size of OMB or the relative size of OIRA within OMB, then it needs to modify its procedures to maximize public participation, not merely tolerate it, so that public expertise can be used to leverage scarce staff resources.
5. Inscrutable 60-day and 30-day notices.
The PRA and OMB’s Information Collection Rule require agencies to publish so-called “60-day notices” to alert the public to an impending ICR and seek public comment on both burden and practical utility, then publish a “30-day notice” alerting the public to submission of the ICR to OMB. For ICRs connected to rulemaking, this notice is supposed to be included in the preamble to the notice of proposed rulemaking or final rule.
Congress established the requirement for 60-day notices as part of the 1995 Amendments to the law. Previously, public participation had been meager because agencies largely evaded the original law’s public notice and comment requirement by publishing only minimal and inscrutable notice, and doing so very late in the process. The 60-day notice requirement was intended to encourage early public participation. The record shows, however, that early notice alone has not solved the public participation deficit. Whereas 30-day notices were both late and inscrutable under the old regime, since 1995 they have been supplemented by equally inscrutable 60-day notices.
To illustrate this inscrutability, I collected every document identified via the Federal Register’s search utility as a 60- or 30-day notice for the week commencing with the publication of this request for comment (October 27¾November 2). There were 39 hits. I report them in Attachments A (60-day notices) and B (30-day notices).
Several transparency defects were observed. For example, determining whether a notice was a 60- or 30-day notice was obvious in some cases (e.g., it was mentioned in the title) but not in others (e.g., only the deadline for submission of public comments revealed it). The inability to easily identify the category to which a notice belonged is itself evidence of inscrutability. In addition, readers often cannot discern the subject matter of an ICR from the title published in the Federal Register. Potentially informative subject matter was included in the titles of twenty-two 60-day notices; nine notices included no identifying information. Of fifteen 30-day notices, nine included subject matter in the title but six did not. The use of generic, undescriptive titles is deterrent to locating a relevant ICR, which inherently impedes public participation.
In my sample, many notices did not clearly disclose whether a response was voluntary or mandatory, even though the explicit disclosure of this information is clearly required by OMB’s Information Collection Rule. Public participation can be expected to be less intense for voluntary ICRs, but agencies’ failure to clearly make this distinction also impedes public participation.
6. Missing, unavailable, or uninformative supporting documents.
Several times I have responded to 60-day notices published in the Federal Register announcing a public comment period concerning a survey conducted or sponsored by a federal agency. At the time of publication, every document necessary for evaluating the proposed ICR must be made public.
In practice, however, a commonplace experience is that all relevant information is not disclosed. Agencies publish 60-day notices inviting public comment on ciphers. When they later report to OMB that they received no comments, the OIRA staff incorrectly interprets this as evidence of a lack of public concern or controversy when in fact it reflects public cluelessness.
Agencies that publish noncompliant 60-day notices violate both the spirit and the letter of the law, and the Information Collection Rule. However, OIRA’s current procedures provide no recourse. OIRA could (but does not) disapprove ICRs for which the 60-day notice was noncompliant. I am unaware of any instance in which OIRA has publicly told an agency to start the ICR process over because the agency violated procedures that are supposed to be mandatory.
OIRA lacks the resources to review and approve 60- and 30-day notices before they are published, and the public has no standing to sue an agency for publishing defective notice. For that reason, OIRA should deputize the public, to whom these notices are addressed, to serve as its eyes and ears. Once defective notice is detected and reported, OIRA should preemptively (and publicly) inform the agency that it will disapprove any ICR submitted that relies on it. An Internet-based tip line, such as I suggested in subsection 3 above to address the problem of bootleg information collections, would be simple to implement and provide an efficient and effective way to generate the information OIRA needs to stop the practice of publishing defective notice.
7. Downwardly biased or absent burden estimates.
OIRA’s request for comment shows that it is well aware that agency burden estimates routinely are downwardly biased and not objectively supported. It is clearly in the agencies’ interest to low-ball paperwork burden; they should never be construed as disinterested parties. The problem is that OIRA has no procedures in place to counteract this inherent bias.
It is routine to read Supporting Statements in which burden estimates are based on the subjective opinions or beliefs¾and sometimes, the wishful thinking¾of agency staff, or perhaps the use of English darts. Opinions, beliefs, and guesswork do not substitute for objectively supported estimates, yet OIRA routinely accepts burden estimates that are founded nothing more.
Agencies routinely certify that their burden estimates are objectively supported, even when they simultaneously disclose elsewhere in a Supporting Statement that these certifications are necessarily false. These discrepancies are not corrected. OIRA staff may rely on public comments to learn whether these claims are true, but the lack of public comments¾often the result of inscrutable notice¾reduces the effectiveness of this lever.
My recent experience with USPTO rulemakings, discussed in Section II(A) above, may be unrepresentative of the government at large. Nonetheless, the Patent Office was remarkably successful at hiding from OIRA staff billions of dollars in annual paperwork burden. It maintained this fiction over a period of several years. When initially presented with evidence showing that USPTO had fantastically low-balled paperwork burdens, OIRA declined to act.
In the course of analyzing a different rule, one that the Patent Office persuaded OIRA staff was too minor to warrant any review at all, about $100 million per year in incremental paperwork burdens were estimated and about $250 million in illegal information collection burdens were discovered in the baseline. Even this information was sufficient only to motivate OIRA not to approve the ICR; it was not sufficient to persuade the Office to disapprove it.
Given this remarkable history of OIRA staff disinterest in objectively supported burden estimates, OMB’s request for comment on how to improve burden estimates comes as a welcome surprise. But the starting point for improving burden estimates is for OIRA itself to take the matter seriously, and to give credence to well-supported burden estimates from public commenters.
8. Failure to consult.
The Information Collection Rule requires agencies to consult with affected members of the public before publishing a 60-day notice requesting comment on the agency’s burden estimates and practical utility claims. In the admittedly few cases in which I have been involved in an ICR review, however, no genuine prior consultation ever occurred.
A simple remedy for OIRA is to direct agencies to reveal the identities and contact information for all persons whom they consulted prior to publishing a 60-day notice. Armed with this information, Desk Officers can easily follow up to verify that consultation actually occurred and that the agency is correctly representing its content.
B. Deficiencies in OIRA Practice
Not every problem in the administration of the PRA is the fault of the agencies. There are numerous ways that OIRA itself undermines the purposes of the law, usually by inaction.
1. Failure to enforce its own rules.
Previously I have documented cases from my experience in which OIRA has declined to exercise its authority to disapprove ICRs based on agency noncompliance with any or even all of the procedural requirements in the Information Collection Rule. It also has declined to disapprove ICRs in which the agency has understated paperwork burdens by billions of dollars per year, and when an agency has failed to make any credible case of practical utility. These inactions reflect a path-of-least-resistance approach to paperwork review. Rarely will OIRA staff generate controversy by erroneously approving an ICR that violates the law or the Information Collection Rule. However, OIRA can and does encounter controversy when following the law leads it to disapprove an ICR that has a powerful political patron.
OIRA’s frequent unwillingness to enforce procedural and substantive provisions of the Information Collection Rule undermines respect for the Rule¾at the agencies, to be sure, but also among the public. It signals that OIRA lacks confidence that the Rule is reasonable and proper, and deserves enforcement. If the public had the legal right to enforce agency compliance, OMB’s timidity would not be so important. But the law makes OIRA’s decisions final and not subject to appeal. That means OIRA has a special duty to rigorously enforce the law and the Information Collection Rule. If OIRA does not enforce it no one else can.
It is possible that there has been a slow decline in OIRA’s attention to paperwork matters, as other duties have arisen. A simple step OIRA can take to reverse this slide is to recommit to the public its intention to resume enforcing the law and the Information Collection Rule. What will be harder, and may take a cultural change within the organization, is for OIRA to follow through on such a commitment. Unless this happens, however, the likelihood is remote that PRA implementation will improve.
2. Failure to educate the public about the PRA.
In a recent public comment regarding centralized regulatory oversight, I stated that the Paperwork Reduction Act was the most powerful procedural law that hardly anyone has ever heard of. Agencies are responsible for much of this ignorance because they have scrupulously avoided educating the public. Their strategic negligence is hardly surprising, for the PRA acts as a brake on their unregulated conduct.
What’s surprising is OIRA’s 29-year record of failing to educate the public. Perhaps nowhere else in the federal administrative state is there such a powerful example of an agency that downplays the significance of its regulatory authority and hides it from those who are supposed to be its beneficiaries. The more common problem is that agencies promote their authorities in an exaggerated manner in hopes that doing so will expand their mission. Yet OIRA does nothing at all to inform and educate the public about the PRA, or to encourage the public’s active participation by submitting public comments.
Fixing what ails PRA implementation requires a radical change in OIRA practices with respect to how it engages the public. Instead of secreting themselves safely behind the barriers of the White House complex, OIRA Desk Officers should make themselves known to the public and available for consultation on paperwork issues in their respective zones of responsibility. OIRA should publish a Citizen’s Guide to the Paperwork Reduction Act and direct agencies to send a copy to every recipient of an information collection request. OIRA should establish a live chat line enabling the public to contact an OIRA professional staff member during regular business hours. OIRA should use social media to spread the word about the PRA and develop a coterie of external experts willing and able to supplement the assistance provided by OIRA staff, much like software makers are supplemented by technical mavens not employed by the vendor.
The array of educational activities OIRA staff could conduct is vast, and in many cases they would not have to leave their desks. The problem is not identifying reasonable alternatives. It is incredibly ironic that the office responsible for setting information policy for the federal government is itself stuck in the 1980s.
3. Failure to protect the public from illegal information collections.
Both the law and OMB’s Information Collection Rule provide ironclad public protection in cases where an agency pursues an enforcement action for failure to provide information that is not covered by a valid OMB Control Number. However, there are important circumstances in which OIRA undermines the PRA’s public protection provisions.
In the first set of circumstances, an agency has significantly (and often repeatedly) violated the procedural requirements of the Information Collection Rule, but OIRA issues a Control Number anyway. Every time OIRA does this, it weakens the Paperwork Reduction Act by incentivizing further misconduct. One can imagine that the first such violation was perceived as inconsequential. But each violation establishes an informal precedent making it increasingly difficult for OIRA to insist on full compliance and increasingly easy for agencies to cut corners. By allowing agency adherence to procedure slip and slide, OIRA incrementally but perceptibly allows them to undermine the law.
In the second set of circumstances, the ICR concerns an illegal agency demand for information to obtain a public benefit. There is no explicit enforcement action against which a person can assert the public protection defense, and failure to submit information not covered by a valid OMB Control Number will result in a denial of the benefit. While the language of the Information Collection Rule establishes that the public protection provision applies in these cases, it is virtually impossible to assert in practice. To be concrete, the public has no recourse if the Social Security Administration, the U.S. Department of Agriculture, or the Department of State denies an application for Social Security benefits, Food Stamps, or a passport, to someone for failing to supply illegally demanded information.
My review of the paperwork requirements imposed on applicants by the U.S. Patent and Trademark Office similarly indicates that the Office makes vast information demands not covered by valid OMB Control Numbers. Yet there is no clear way for applicants to invoke their public protection rights. Because they are seeking a public benefit¾in this case, one that the government is required by law to provide unless it proves the applicant is not entitled to receive it¾applicants have no protection at all from illegal information demands made by USPTO patent examiners.
OIRA has a special duty in these cases to protect the public’s legal right not to respond to illegal information collections. There are many ways it could do so. For example, it could issue advisory opinions that alert the public to illegal information collections, inform them of their legal rights, and provide an OIRA point of contact for the submission of complaints. Agencies likely would respond very quickly, thus reducing the number of advisory opinions unresolved to near zero.
4. Failure to conduct reviews transparently.
Unlike OIRA’s review of draft proposed and final regulations, which is conducted deliberatively pursuant to presidential authority, OIRA’s review of ICRs is supposed to be part of a public process. Reconciling these opposing schemes has been difficult for OIRA, and especially so in cases where a regulation also contains paperwork burdens. In these cases, a system has not been devised that would adequately open the paperwork review to the public while maintaining confidentiality during regulatory review.
This conflict does not arise for ICRs that do not involve rulemakings. Nonetheless, the OIRA staff tend to follow analogously restrictive practices regarding public participation. Although OIRA claims to fully and promptly disclose all relevant information related to ICRs, in fact it does not. Contrary to the practice of Regulations.Gov, the government-wide website that OMB demands other agencies use to permit interactive public participation, OMB hamstrings its own website at Reginfo.Gov to prevent the public from viewing comments submitted in response to 30-day notices. Only after OIRA has concluded its review does it make these comments public. If there is any legitimate reason for suppressing the disclosure of public comments in real time, OIRA has not explained what it might be.
OMB also suppresses the disclosure every version of a Supporting Statement except the final one that forms the basis for its decision. During the course of ICR review, changes are often made, such as to improve burden estimates or make other important modifications. Supporting Statement updates are not announced, either by OIRA or the authoring agency. Only those members of the public who know where and when to look are able to keep themselves informed. There is no justification for such secrecy.
Finally, OIRA suppresses the disclosure of agency responses to public comments it receives from the public on 30-day notices. During OMB’s review of EPA’s EDSP Tier 1 Test Order ICR, senior EPA staff acknowledged at a public meeting that the Agency had responded to public comments sent to OIRA. But EPA did not post them on Regulations.Gov and OIRA did not disclose them on Reginfo.Gov so that public commenters could offer rebuttal. OIRA also has declined to make these responses public on request. When the Freedom of Information Act provides the only legal way to gain access to relevant information, an agency’s administrative process cannot be regarded as transparent or trustworthy.
OIRA’s lack of transparency is fundamentally incompatible with the spirit, if not the letter, of the PRA and OMB’s Information Collection Rule. It frustrates effective public participation and creates significant doubt about OIRA’s seriousness.
5. Failure to effectively utilize public comments.
I have documented numerous reasons why the public does not participate in ICR reviews as actively as it should. Similarly problematic is OIRA’s limited attention to for the comments it does receive. To be concrete, I am unaware of any instance in which OIRA supplanted biased or unsupported burden estimates submitted by an agency with better-supported estimates provided by a public commenter.
This is a significant missed opportunity. OIRA Desk Officers should compare the quality of burden estimates they receive from public commenters and the agency, and select the better of the two. This would be a powerful tool for improving the quality of burden estimates.
6. Minimal attention to information quality.
OMB’s information quality guidelines directed agencies to inculcate information quality principles throughout their operations. Virtually all agencies committed in 2002 to do so; some agencies may have actually done so; most have ignored even their most basic procedural commitments. Today, OIRA requires agencies to certify compliance with their information quality guidelines in every ICR they submit. Agencies dutifully provide these certifications, but they provide no evidence of actual compliance. Oftentimes it is so easy to discern information quality violations in a Supporting Statement that it is inconceivable that the person certifying compliance has any clue what information quality means.
Agencies feign compliance because that is all OIRA asks from them. When public commenters include information quality complaints in their public comments, it appears that OMB isn’t listening. Few OIRA staff members seem to be well versed in information quality principles or OMB’s own guidelines.
This sad state of affairs is deeply disturbing. Although OMB did not get around to issuing government-wide guidelines until it was compelled by law to do so by 2002, information quality is the sine qua non of the Paperwork Reduction Act. Yet in OMB’s request for comments on how to improve its implementation of the PRA, the importance of information quality in improving PRA implementation is nowhere to be found.
7. Failure to verify adherence to information collection protocols or Terms of Clearance.
Whether OIRA enforces the Terms of Clearance in EPA’s EDSP Tier 1 Test Order ICR is just the tip of a very large iceberg. With only the rarest exceptions, once OIRA approves an ICR the transaction id completed and it never revisits the matter to determine whether the agency implemented the ICR in the manner set forth in the Supporting Statement. If the agency promised to use a probability sample for a survey, OIRA doesn’t find out if it actually did so. If an agency committed to achieve an 80% or better response rate, OIRA doesn’t learn if it succeeded. If there were Terms of Clearance setting forth restrictions on the extrapolation of data, OIRA has no system in place for detecting, much less preventing, the data from being misused.
One of the most effective things OIRA could do to improve the implementation of the PRA is to establish systems permitting verification of adherence to Supporting Statements and Terms of Clearance. For example, whenever an agency disseminates information collected pursuant to a valid OMB Control Number, it should be required to include links to all ICR documents and Terms of Clearance and a way to contact OIRA in case discrepancies are found. This would enable the public to do the validation that OIRA lacks sufficient resources to do. If combined with an expedited path to prompt error correction, the incidence of material departures would decline greatly.
C. Deficiencies in Public Participation
Public participation in PRA reviews has not lived up to expectations for several reasons. First, as noted above, it is commonplace for agencies to publish inscrutable 60- and 30-day notices. These notices consist of boilerplate recitations of various provisions of the PRA or OMB’s Information Collection Rule that few members of the public understand, and little information that the public might find helpful for informing public comment. Burden estimates are presented as ranges and point estimates containing a mix of precise and arbitrary values. To be concrete, in one of the USPTO 60-day notices mentioned in Section II(A), unit burden estimates were comically reported as ranging from “1 minute and 48 seconds to 12 hours” per response, with exactly (!) 2,284,439 responses. No estimate was given of the incremental burden likely to arise from the proposed rule. USPTO, like most other agencies, erects barriers to informed public participation by providing the least amount of information OIRA will accept in a form that the public is least able to use.
Second, most agencies make no effort to educate the public concerning how to participate effectively. The Information Collection Rule requires agencies to consult with affected members of the public before seeking public comment, and agencies nearly always certify in their submissions to OMB that they have done so. Identifying any person who has been actually consulted is a daunting task.
As noted in Section III(B)(2), OIRA’s own public education effort is even more desultory than that of the agencies. Indeed, it hard not to draw the inference that OIRA is institutionally even less enthusiastic about public participation than the agencies it oversees, and that OIRA is satisfied when agencies perform the bare minimum.
Third, as noted in Section III(B)(6), supporting information that agencies are supposed to make available simultaneously with a 60-day notice often is devilishly hard to locate or does not actually exist. I have personally experienced instances in which I was told that supporting information I needed to file informed comments would not be available until after the public comment period had conveniently expired.
Finally, in the rare case in which active public participation has occurred in response to a 60-day notice, there is little evidence that agencies responded to these comments in their submissions to OMB, even though the Information Collection Rule requires such responses. A public participation process that the public perceives as ineffective or empty is not one that will motivate much public participation.
IV. ADDITIONAL COMMENTS ON IMPROVING IMPLEMENTATION OF THE PAPERWORK REDUCTION ACT
A. Lessons Already Learned
In Section I, I discussed nine recommendations for reform that had been made by others prior to OMB’s request for comment. I showed why seven of them were self-serving, lacked objective merit, or were otherwise unjustified. The two survivors have nothing in common; one (meaningful cash incentives for survey participation) would improve response rates and the other (ICR/regulation tracking) would make the PRA process more accessible to the public.
In Section II, I provided examples from recent experience that, in my view, illustrate how extensive are the problems that currently plague OMB’s implementation of the PRA. The series of USPTO regulations and ICRs shows an agency committed to evading OIRA oversight on both regulatory and paperwork review. Because the agency was so successful, it also casts a dark shadow on OIRA’s performance. USPTO misrepresentations appear to have begun at the beginning¾that is, in the Patent Office’s initial Regulatory Agenda entries, which made false statements about the scope, scale, and effects of these rules. When USPTO said it intended to “revise the rules of practice to share the burden of examining an application,” it was an early sign that it intended to impose massive increases in paperwork burdens. OIRA apparently missed these early signals and ignored later alarms. But for the action of the U.S. District Court of the Eastern District of Virginia, which issued a permanent injunction against the combined Claims and Continuations Practice final rule, OIRA would have had the choice of booking about $30 billion in incremental, discretionary annual paperwork burdens to the $1.7 billion information collection budget of the Department of Commerce¾or, alternatively, cynically disregard the PRA.
The case of EPA’s EDSP Tier 1 Test Order ICR underscores three especially difficult PRA implementation challenges. First, while there is no dispute that EPA has sufficient statutory authority to require the submission of test data, this authority is not unbounded. EPA clearly cannot use its authority to compel the public to submit information that isn’t germane to the EDSP. Its statutory authority is not a blank check authorizing it to demand anything at all. Similarly, but less obviously, it cannot use its statutory authority to force the submission of data that do not materially advance its implementation of the law for which the authority to collect information was created.
Second, agencies’ authority to impose mandatory information collections is constrained by limits in the competing authority delegated to OMB via the Paperwork Reduction Act to minimize burdens on the public. Although OIRA cannot legally interfere with EPA’s exercise of substantive authority under the Food Quality Protection Act, it likewise cannot legally approve information collections that do not have practical utility.
Third, the approach OIRA has taken to finesse these dilemmas through Terms of Clearance will fail to defend the integrity of the PRA unless the Office ensures agency compliance. OIRA’s Terms of Clearance include conditions without which its approval of the ICR arguably would have been contrary to law. OIRA’s defense of the PRA thus depends entirely on its willingness and ability to enforce its Terms of Clearance. The public, whom OIRA is statutorily charged with defending against overly burdensome information collection, cannot do so on its own. It lacks access to Article III courts, and OIRA itself has no administrative procedures in place that provide an alternative appellate venue to hear complaints of agency misconduct.
The case of the Census Bureau renting out to EPA its statutory authority to make an information collection mandatory is an example of citizen abuse for which OIRA has an easy remedy. OIRA can, as I recommended in Section II(B)(3), audit the agencies to find out the prevalence of such abuse. It can refuse to approve future ICRs in which this practice arises, and it can exercise its authority under the Information Collection Rule to reopen any currently approved ICR in which statutory is now being rented out, and put a stop to it.
B. Better Utilizing the PRA Process to Improve the Quality of Regulation
In comments I submitted to OMB on behalf of Regulatory Checkbook regarding presidential regulatory review, I recommended that the Administration:
breathe new life into the PRA and convert it from a sleepy procedural statute that hardly anyone has heard of and make it the dominant workhorse for the production of high‐quality data, well‐constructed and transparent models, and enriched benefit‐cost analyses that make the extraordinarily difficult job of governing just a little bit easier.
Specifically, I called for separating the data collection and data analysis components of regulatory oversight¾chiefly, the preparation of Regulatory Impact Analyses and their subordinate components¾from the policy deliberative aspects of the formal review process. OMB should:
direct agencies to establish and follow a public process, initiated long before a draft proposed rule is written, to scope, design, and structure the regulatory analysis that will be used to inform decision-making. The products of this process¾Regulatory Analysis Blueprints¾would permit a vibrant and civil public discussion about how best to proceed. It would enable all parties to ensure that the alternatives they care about most are identified early and included in the analysis. If there are questions about data or analytic methods, raising these questions early will improve the quality of analysis and significantly enhance the transparency of the entire regulatory process. Any outside party could choose to perform a shadow RIA, thus creating external pressure on the agency to take quality very seriously (p. 55).
Regulatory Analysis Blueprints are an innovation that originated at EPA. Whereas the EPA model includes policy matters as well as analytics, and was not shared with the public, OIRA could build a model that is strictly limited to science, economics, and regulatory analysis, and which could involve the public at every step of the process. This would help reconcile a longstanding conundrum: How can the OIRA fulfill its statutory responsibility under the Paperwork Reduction Act, which requires that it establish and cultivate regular contacts with the public, with the virtual prohibition on ex parte communication that must be applied during regulatory review? The answer is to move the analytic components of regulation into the Paperwork Reduction Act process.
In my comments I said that the Paperwork Act “may be the most important procedural law that hardly anyone has heard of.” Adopting this recommendation would go a long way toward ending public ignorance. It would allow OIRA staff to play a much more constructive role, using the PRA process to supply the market for high quality information that regulatory decision-making and other legitimate government functions require.
The reactive model of PRA implementation that OIRA follows was a reasonable approach to the conditions that existed in 1980. It no longer serves the public interest. It is time to adopt a proactive model, one that anticipates the legitimate information needs of the government, while energizing the public to participate fully in the process to hold agencies accountable for ensuring quality and minimizing burden.
* * *
Thank you for the opportunity to provide these comments. I look forward to the opportunity to assist you in every effort to improve the implementation of the Paperwork Reduction Act, and to transform it into an effective tool for the advancement of good government that it was intended to be.
RICHARD B. BELZER
Attachment A: Transparency in 60-Day Notices, 10/27/09–11/02/09
Attachment B: Transparency in 30-Day Notices, 10/27/09 – 11/02/09
Arbuckle DR. 2006. Memorandum for Selected Chief Information Officers; Reaching Full Compliance with the Paperwork Reduction Act. Available: http://www.whitehouse.gov/omb/assets/omb/inforeg/pra_memo_to_cio_051006.pdf [accessed December 28 2009].
Bass GD, Bird M, DeWaal CS, Duthu NB, Goldston DJ, Greenwood M, Grifo F, Irons J, Jayne ES, Johnson S, Michaels D, Parker RW, Radin B, Rushing R, Shull JR, Stauss PL, Warren W. 2008. Advancing the Public Interest Through Regulatory Reform. Washington, D.C.: OMB Watch.
Belzer RB. 2002. Exposure Assessment at a Crossroads: The Risk of Success. Journal of exposure analysis and environmental epidemiology 12: 96-103.
Belzer RB. 2007. Cost of Complying with the Proposed IDS Rule; Meeting at OMB, October 18, 2007. Available: http://www.reginfo.gov/public/do/DownloadDocument?documentID=57760&version=1 [accessed December 21 2009].
Belzer RB. 2008a. Letter to Nicholas A. Fraser, Desk Officer for the U.S. Patent and Trademark Office, Office of Information and Regulatory Affairs
Office of Management and Budget RE: ICR 0651-00xx. Available: http://www.reginfo.gov/public/do/DownloadDocument?documentID=90554&version=1 [accessed December 28 2009].
Belzer RB. 2008b. Letter to Nicholas A. Fraser, Desk Officer for the U.S. Patent and Trademark Office, Office of Information and Regulatory Affairs
Office of Management and Budget RE: ICR 0651-00xx: ICs and Burden Estimates. Available: http://www.reginfo.gov/public/do/DownloadDocument?documentID=93894&version=1 [accessed December 28 2009].
Belzer RB. 2008c. Letter to Susan E. Dudley, Administrator, Office of Information and Regulatory Affairs, Office of Management and Budget RE ICR 0651-0031. Available: http://www.reginfo.gov/public/do/DownloadDocument?documentID=57744&version=1 [accessed December 21 2009].
Belzer RB. 2008d. Risk Assessment and Information Quality. “Society for Risk Analysis, 2008 Annual Meeting”. Boston, Mass.
Belzer RB. 2009a. An Analysis of EPA’s Information Collection Request Seeking OMB Approval to Impose Mandatory Tier 1 Assay Testing in Support of the Endocrine Disruptor Screening Program,” Attachment 1 to: Ferenc, S., Siddiqui, I., Fratz, D., Walls, M., Letter to the Office of Information and Regulatory Affairs, Office of Management and Budget Re: EPA Information Collection Submission To OMB for Review and Approval; Tier 1 Screening of Certain Chemicals Under the Endocrine Disruptor Screening Program (EDSP); EPA ICR No. 2249.01, OMB Control No. 2070-New Docket ID Number EPA-HO-OPPT-2007-1081, . Available: http://www.regulations.gov/search/Regs/home.html#documentDetail?R=09000064809c90ed [accessed December 21 2009].
Belzer RB. 2009b. Presidential Regulatory Review: Suggestions for Reform. Available: http://www.reginfo.gov/public/jsp/EO/fedRegReview/Regulatory_Checkbook_Comments_-_Belzer.pdf [accessed December 15 2009].
Boundy DE. 2007. Letter to Susan E. Dudley, Administrator, Office of Inforination and Regulatory Affairs, Office of Management and Budget, RE RIN: (0651-AB-93 (“Changes to Practice for Continuing Applications, Requests for Continued Examination Practice, and Applicationsd Applications Containing Patentably lndistiilct Claiins” and RIN: 0651-AB-94 (“Changes to Practice for the Exaininatioil of Claiins in Patent Applications”). Available: http://www.whitehouse.gov/omb/assets/omb/oira/0651/meetings/619-1.pdf (“Enclosyre 1,” cover letter), http://www.whitehouse.gov/omb/assets/omb/oira/0651/meetings/619-2.pdf (“Enclosure 2,” signatories), http://www.whitehouse.gov/omb/assets/omb/oira/0651/meetings/619-3.pdf (“Enclosure 3,” attachments) [accessed December 22 2009].
Graham JD. 2002a. Memorandum for Chief Information Officers, General Counsels, and Solicitors; Ensuring Full Compliance with the Paperwork Reduction Act. Available: http://www.whitehouse.gov/omb/assets/omb/inforeg/violation_memo060602.pdf [accessed December 28 2009].
Graham JD. 2002b. Memorandum for Chief Information Officers; Ensuring Full Compliance with the Paperwork Reduction Act. Available: http://www.whitehouse.gov/omb/assets/omb/inforeg/pra_memo111302.pdf [accessed December 28 2009].
Graham JD. 2003. Memorandum for the Chief Information Officers; Ensuring Full Compliance with the Paperwork Reduction Act. Available: http://www.whitehouse.gov/omb/assets/omb/inforeg/cio_memo053003.pdf [accessed December 28 2009].
Graham JD, Lefkowitz JP. 2001. Memorandum for Chief Information Officers, General Counsels, and Solicitors; Ensuring Full Compliance with the Paperwork Reduction Act. Available: http://www.whitehouse.gov/omb/assets/omb/inforeg/pra_memo111401.pdf [accessed December 28 2009].
Graham JD, Newstead J. 2004. Memorandum to Chief Information Officers, General Counsels, and Solicitors; Ensuring Full Compliance with the Paperwork Reduction Act. Available: http://www.whitehouse.gov/omb/assets/omb/inforeg/compliance_pra092704.pdf [accessed December 28 2009].
Graham JD, Newstead J. 2005. Memorandum for Selected Chief Information Officers, General Counsels, and Solicitors; Ensuring Full Compliance with the Paperwork Reduction Act. Available: http://www.whitehouse.gov/omb/assets/omb/inforeg/pra_full_compliance_032105.pdf [accessed December 28 2009].
Harrington W, Heinzerling L, Morgenstern RD, eds. 2009. Reforming Regulatory Impact Analysis. Washington, D.C.: Resources for the Future.
ICF International. 2007. Certification Analysis under the Regulatory Flexibility Act; Changes to Practice for Continued Examination Filings, Patent Applications Containing Patentably Indistinct Claims, and Examination of Claims in Patent Applications. 1-37.
Johnson III C, Graham JD. 2004. Memorandum for President’s Management Council; Ensuring Full Compliance with the Paperwork Reduction Act. Available: http://www.whitehouse.gov/omb/assets/omb/inforeg/compliance_pra_memo092704.pdf [accessed December 28 2009].
Katznelson R. 2008. Letter to Nicholas A. Fraser, Desk Officer for U.S. Patent and Trademark Office, Office of Information and Regulatory Affairs,
Office of Management and Budget. Available: http://www.reginfo.gov/public/do/DownloadDocument?documentID=95757&version=1 [accessed December 28 2009].
Mage DT, Wallace L, Kollander M, Ott WR. 2006. Statistical Issues in Farmworker Studies. Envionmental Health Perspectives, 114(12): A688-A689.
Markey EJ. 2009. Letter to Peter Orszag, Director, Office of Management and Budget. Available: http://markey.house.gov/docs/letteromb102209.pdf [accessed.
McGarity TO. 1991. Reinventing Rationality. Cambridge, U.K.: Cambridge University Press.
Miller III JC. 2006. Prepared Statement of James C. Miller III before the Subcommittee on Federal Financial Management, Government Information, and International Security of the Committee on Homeland Security and Governmental Affairs of the United States Senate. Available: http://hsgac.senate.gov/public/index.cfm?FuseAction=Files.View&FileStore_id=7b9e561f-6bc7-47b7-b9a5-4cde799f10d0 [accessed December 27 2009].
Mosquin P, Whitmore R, Suerken C, Quackenboss J. 2005. Population Coverage and Nonresponse Bias in a Large-Scale Human Exposure Study. Journal of exposure analysis and environmental epidemiology 15: 431-438.
Office of Management and Budget. 1990. Current Regulatory Issues in Risk Assessment and Risk Management. In: Regulatory Program of the United States, April 1, 1990 — March 31, 1991. Washington, DC: Office of Management and Budget, 13-26.
Office of Management and Budget. 2002. Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies; Notice; Republication. Federal Register 67(36): 8452-8460.
Office of Management and Budget. 2005. Final Information Quality Bulletin for Peer Review. Federal Register 70(10): 2664-2667.
Office of Management and Budget. 2006. Standards and Guidelines for Statistical Surveys. Available: http://www.whitehouse.gov/omb/inforeg/statpolicy/standards_stat_surveys.pdf [accessed October 1 2007].
Office of Management and Budget. 2008. National Children’s Study Formative Generic Clearance. Available: http://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=200802-0925-003 [Supporting Statement, Part A]; http://www.reginfo.gov/public/do/DownloadDocument?documentID=60495&version=1 [Supporting Statement, Part B]; http://www.reginfo.gov/public/do/DownloadDocument?documentID=60494&version=1 [Appendix A: Questionnaire] [accessed December 26 2009].
Office of Management and Budget. 2009. Improving Implementation of the Paperwork Reduction Act. Federal Register 74(206): 55269-55272.
Orszag PR. 2009. Letter to Rep. Edward J. Markey [ICR 2070-0176]. Washington, D.C.
U.S. Census Bureau. 2008. Pollution Abatement Costs and Expenditures: 2005 [MA200(05)]. Washington, D.C.: U.S. Census Bureau, U.S. Department of Commerce.
U.S. Environmental Protection Agency. 1999a. Proceedings of the NHEXAS Data Analysis Workshop. Available: http://www.epa.gov/nerl/research/nhexas/nhexas.pdf [accessed December 14 2009].
U.S. Environmental Protection Agency, 1999b, The Benefits and Costs of the Clean Air Act, 1970 to 1990. Agency USEP, Secondary The Benefits and Costs of the Clean Air Act, 1970 to 1990.
U.S. Environmental Protection Agency. 2000. Guidelines for Preparing Economic Analyses. “EPA 240-R-00-003”. Washington, D.C.: U.S. Environmental Protection Agency.
U.S. Environmental Protection Agency. 2008. Guidelines for Preparing Economic Analyses: External Review Draft. Washington, D.C.: U.S. Environmental Protection Agency.
U.S. Environmental Protection Agency. 2009. Agency Information Collection Activities; Submission To OMB for Review and Approval; Comment Request; Tier 1 Screening of Certain Chemicals Under the Endocrine Disruptor Screening Program (EDSP); EPA ICR No. 2249.01, OMB Control No. 2070–New. Federal Register 74(71): 17477-17479.
U.S. Patent and Trademark Office. 2005a. Regulatory Agenda #565. Changes to Rules of Practice Before the Board of Appeals and Interferences in Ex Parte Appeals [0651-AC12]. Federal Register 72(82): 22423.
U.S. Patent and Trademark Office. 2005b. Regulatory Agenda #741. Changes to Practice for Continuing Applications, Requests for Continued Examination Practice, And Applications Containing Patentably Indistinct Claims [RIN 0651-AB93]. Federal Register 70(209): 64479.
U.S. Patent and Trademark Office. 2005c. Regulatory Agenda #742. Changes to Practice for the Examination of Claims in Patent Applications [RIN 0651-AB94]. Federal Register 70(209): 64479.
U.S. Patent and Trademark Office. 2005d. Regulatory Agenda #743. Changes to Information Disclosure Statement Requirements and Other Related Matters [RIN 0651-AB95]. Federal Register 70(209): 64479-64480.
U.S. Patent and Trademark Office. 2006a. Changes To Information Disclosure Statement Requirements and Other Related Matters; Proposed Rule [0651-AB95]. Federal Register 71(131): 38808-38823.
U.S. Patent and Trademark Office. 2006b. Changes To Practice for Continuing Applications, Requests for Continued Examination Practice, and Applications Containing Patentably Indistinct Claims; Proposed Rule [0651-AB93]. Federal Register 71(1): 48-61.
U.S. Patent and Trademark Office. 2006c. Changes to Practice for the Examination of Claims in Patent Applications; Proposed Rule [0651-AB94]. Federal Register 71(1): 61-69.
U.S. Patent and Trademark Office. 2006d. Regulatory Agenda #584. Examination of Patent Applications That Include Claims Containing Markush or Other Alternative Language [0651-AC00]. Federal Register 71(78): 22461-22462.
U.S. Patent and Trademark Office. 2007a. Changes To Practice for Continued Examination Filings, Patent Applications Containing Patentably Indistinct Claims, and Examination of Claims in Patent Applications; Proposed Rule [0651-AB93, 0651-AB94]. Federal Register 72(161): 46716-46814.
U.S. Patent and Trademark Office. 2007b. Examination of Patent Applications That Include Claims Containing Alternative Language; Proposed Rule [0651-AC00]. Federal Register 72(154): 44992-45001.
U.S. Patent and Trademark Office. 2007c. Rules of Practice Before the Board of Patent Appeals and Interferences in Ex Parte Appeals; Proposed Rule [0651-AC12]. Federal Register 72(145): 41472-41490.
U.S. Patent and Trademark Office. 2007d. Submission for OMB Review; Comment Request [0651-0031]. Federal Register 72(180): 53232-53233.
U.S. Patent and Trademark Office. 2008a. Board of Patent Appeals and Interferences Actions; Information Collection Request; New Action; 60-day Notice. Federal Register 73(111): 32359-32561.
U.S. Patent and Trademark Office. 2008b. Rules of Practice Before the Board of Patent Appeals and Interferences in Ex Parte Appeals; Final Rule [0651-AC12]. Federal Register 73(112): 32938-32977.
U.S. Patent and Trademark Office. 2008c. Submission for OMB Review; Comment Request [0651-00xx, AC12]. Federal Register 73(196): 58943-58944.
U.S. Patent and Trademark Office. 2009a. Rules of Practice Before the Board of Patent Appeals and Interferences in Ex Parte Appeals; Delay of Effective and Applicability Dates. Federal Register 73(238): 74972.
U.S. Patent and Trademark Office. 2009b. Rules of Practice Before the Board of Patent Appeals and Interferences in Ex Parte Appeals; Request for Comments on Potential Modifications to Final Rule and Notice of Roundtable During Comment Period. Federal Register 74(244): 67987-68004.
U.S. Patent and Trademark Office. 2009c. Supporting Statement, United States Patent and Trademark Office, Board of Patent Appeals and Interferences (BPAI) Actions: OMB Control Number 0651-00xx (December 3, 2009 Version). Available: http://www.reginfo.gov/public/do/DownloadDocument?documentID=89627&version=2 [accessed December 28 2009].
Viscusi WK, Magat WA, Huber J. 1991. Pricing Environmental Health Risks: Survey Assessments of Risk-Risk and Risk-Dollar Trade-Offs for Chronic Bronchitis. Journal of Environmental Economics and Management 21: 32-51.
 Office of Management and Budget (2009).
 Belzer (2009b).
 Office of Management and Budget (2009, p. 55271-55272).
 44 U.S.C. § 3501(1).
 44 U.S.C. § 3501(2).
 44 U.S.C. § 3501(4).
 Harrington et al. (2009, pp. 231-233).
 Nothing in the Harrington et al. discussion suggests that OMB’s “intrusions” were politically motivated. Rather, the authors consistently imply that OMB insisted on quality standards that “many researchers” did not want to meet. A similar complaint was made years ago by McGarity (1991, pp. 283-284), though with the charge that OIRA’s actions were indeed politically motivated. McGarity’s evidence consisted of claims made by unnamed sources in a few articles in the trade press, a common venue used by agency employees to make anonymous claims and complaints.
 Viscusi et al. (1991).
 Convenience samples are generally prohibited for the estimation of population effects because their representativeness to any population cannot be determined. See Office of Management and Budget (2006, Standard 1.2). Nonetheless, results from convenience samples often are incorrectly extrapolated to populations as if they were representative. Compounding this error, when results are disseminated the fact that they are the product of convenience sampling is rarely mentioned.
An “energetic” defense of probability sampling in environmental health, and a rejection of convenience sampling, can be found in a recent letter to the editor of a prominent scientific journal by two retired EPA scientists and their colleagues (Mage et al. 2006). They explicitly criticize other published work for failing to adhere to OMB statistical policy standards that EPA has publicly embraced, and express disappointment that, despite this obvious violation, the original research passed the Agency’s internal review procedures. Harrington et al. seek an exemption from these statistical policy standards for WTP surveys, thus implying that the estimation of benefits from reducing health risks need not be so careful as the estimation of the risks themselves.
 Viscusi et al. (1991, p. 50)
 See, e.g., U.S. Environmental Protection Agency (1999b, p. I-4 [CAA Section 812, 1970-1990]) and (2000, p. 95 [EPA Guidelines for Regulatory Impact Analysis]).
 Harrington et al. (2009, p. 25).
 Harrington et al. (2009, cover letter at 2).
 See, e.g., the discussion of EPA’s National Human Exposure Assessment Survey (NHEXAS) in Section I(F) below.
 To take an obvious example in which data exist, during the Bush 43 administration OIRA reviewed 5,125 draft regulations and returned just 33. It defies reason to infer that only 0.6% of all draft rules failed to adhere to the regulatory principles of Executive Order 12,866, or that 99.4% adhered to the anti-regulatory philosophy routinely ascribed to George W. Bush.
 Changes in an information collection often are made after OMB approval of an ICR. A rule of reason should prevail for deciding whether these changes in fact result in material defects. Sometimes, changes improve the quality of an information collection without materially increasing burden, and it would be unreasonable to penalize an agency for adapting to new information. On the other hand, changes that increase burden without a demonstrable improvement in practical utility must not be condoned. Similarly, when a survey designs promises to achieve an 80% response rate but does not do so in practice, the data must not be disseminated without punishingly strong disclaimers unless the agency can persuasively demonstrate the absence of nonresponse bias, as required by OMB statistical policy standards (Office of Management and Budget 2006).
 The choice of evidentiary standard is crucial for this reform to work. A relatively weak standard, such as preponderance of the evidence, would incentivize cheating. Similarly, ambiguity about the applicable evidentiary standard also makes failure likely. A key defect in OMB’s government-wide information quality guidelines is the rebuttable presumption of “adequate” quality for information that has been “peer reviewed.” Because OMB’s guidelines do not specify the applicable evidentiary standard, the public has no credible way to know what is required to successfully rebut. See Office of Management and Budget (2002, p. 8459 (“rebuttable based on persuasive showing … in a particular instance”)). OMB’s peer review guidelines exacerbate this defect by failing to establish objective metrics for the “adequacy” of peer review (Office of Management and Budget 2005, p. 2675 (“adequacy” of peer review is determiend subjectively and nontransparently)). Peer review never should provide a presumption of information quality if, for example, information quality principles were not key elements of the peer review charge fully addressed by the reviewers. Astonishingly, OMB’s peer review guidelines permit precisely this sort of corruption, and they reward it if it occurs under the auspices of the National Research Council by turning the NRC into a virtual scientific Curia.
 The time committed to public comment apparently is not their concern. They suggest, using a peculiar double-negative construction, that “it might not be inappropriate for OMB to request reviews from qualified professionals, or to invite commentary from all members of relevant scientific disciplines.” See Harrington et al. (2009, p. 232). This would take time, and probably more time than public comment.
 My OMB tenure ended in 1998, so it is possible that all WTP surveys submitted to OIRA since that date pass these elementary tests.
 Comments spun up on behalf of researchers are a special form of the “interest group” comments that Harrington et al. dismiss. Because they lack independence, they deserve greater technical scrutiny rather than deferential treatment.
 McGarity (1991) used a similar approach, citing the opinions of unnamed “agency analysts” and “critics” as the basis for concluding that, “in the minds of some,” OIRA misuses the PRA to “inhibit agency rulemaking indirectly” (pp. 283-284).
 It is hard to make a cogent argument that improving the response rate leads to bias. For OIRA’s position to be statistically valid, it would have to be the case that the provision of cash incentives motivates respondents to give biased answers. This can only be true if respondents believe that the payment of the incentive is subtly conditioned on the nature of the response. This circumstance could occur, of course, but a survey that leads respondents to develop such perceptions has systematic defects other than a low response rate.
 Provided that the sample was representative, the practical utility of NHEXAS was obvious: EPA would be able to estimate human risk using human exposure data instead of relying on biased default assumptions.
 See U.S. Environmental Protection Agency (1999a, p. 116). An additional $75 was paid to those who completed the duplicate diet component, but the response rate for that component was not provided. Over the course of the study, response rates declined: Visit 1 (80%), Visit 2 (57%), and Visit 3 (48%).
 Office of Management and Budget (2006, Guidelines 1.3.4 and 3.2.9).
 Mosquin et al. (2005).
 EPA’s 2000 guidance on preparing Regulatory Impact analyses does not mention response rates or establish any performance standard. Response rates are discussed only in passing in the Agency’s 2008 draft revision. See U.S. Environmental Protection Agency (2000; 2008, p. 7-37).
 Bass et al. (2008, p. 26).
 See, e.g., Miller III (2006).
 Some agencies, most notably the Environmental Protection Agency, includes OMB Control Numbers within its regulations as published in the Code of Federal Regulations. This enables a crosswalk from regulation to ICR, but not in reverse.
 Table 1 below lists these ICRs alongside the regulatory actions to which they apply, and summarizes the major deficiencies in USPTO’s compliance with both the Paperwork Reduction Act and Executive Order 12,866.
 (a) “Changes To Information Disclosure Statement Requirements and Other Related Matters (U.S. Patent and Trademark Office 2006a); (b) “Examination of Patent Applications That Include Claims Containing Alternative Language; Proposed Rule” (U.S. Patent and Trademark Office 2007b); and (c) “Rules of Practice Before the Board of Patent Appeals and Interferences in Ex Parte Appeals; Proposed Rule” (U.S. Patent and Trademark Office 2007c).
 (a) “Changes To Practice for Continuing Applications, Requests for Continued Examination Practice, and Applications Containing Patentably Indistinct Claims; Proposed Rule [0651-AB93]” (U.S. Patent and Trademark Office 2007a); and (b) “Changes to Practice for the Examination of Claims in Patent Applications; Proposed Rule [0651-AB94]” (U.S. Patent and Trademark Office 2006c). Generally, a draft rule is economically significant (Executive Order 12,866) and major (5 U.S.C. 802(2)) if it is likely to result in effects of $100 million or more in any one year.
 USPTO (2007a, p. 46434): “This rule making has been determined to be significant for purposes of Executive Order 12866 (Sept. 30, 1993), as amended by Executive Order 13258 (Feb. 26, 2002) and Executive Order 13422 (Jan. 18, 2007).”
 Determining whether these changes would have had net social benefits requires the preparation of a competently performed Regulatory Impact Analysis. Through its false certifications and OIRA’s lack of interest in validating them, USPTO successfully evaded the requirement to prepare any RIAs. And the Patent Office continues to view Executive Order 12,866 as containing procedures and requirements to which it is exempt. In the December 2009 advance notice of proposed rulemaking through which it hopes to extricate itself from a mess of its own creation, the Patent Office ignores Executive Order 12,866 and gives no estimates of costs, benefits, or paperwork burden¾even for the specific changes in regulatory language that it says it is considering (U.S. Patent and Trademark Office 2009b).
 In one case, USPTO issued the 60-day notice one day before promulgating the final rule to which it applied. See USPTO (2007c, [proposed rule]; 2008a, [“60-day notice”]; 2008b, [final rule]). It is difficult to imagine a more cynical attempt to evade accountability.
 Belzer (2008c).
 At least one of USPTO’s certifications was correct in only the most bizarre way: The proposed rule did not add any new burdens to an existing ICR because there was no existing ICR!
 The Information Collection Rule requires agencies to respond to public comments received in response to 60-day notices in the Supporting Statement submitted along with an ICR. See 5 C.F.R. § 1320.5(a)(1)(ii). USPTO’s Supporting Statements consistently ignore or mischaracterize these public comments, a practice that OIRA has consistently tolerated.
 See the 60-day notice, June 9, 2008 (U.S. Patent and Trademark Office 2008a); Final rule, June 10, 2008 (U.S. Patent and Trademark Office 2008b); the 30-day notice, October 8, 2008 (U.S. Patent and Trademark Office 2008c); and notice indefinitely staying the final rule pending resolution of paperwork issues, December 10, 2008 (U.S. Patent and Trademark Office 2009a).
 On procedural noncompliance, see primarily Belzer (Belzer 2008a, 2008b). On substantive noncompliance, see especially Katznelson (2008). All public comments in response to the 30-day notice are online at http://www.reginfo.gov/public/do/PRAViewDocument?ref_nbr=200809-0651-003,
 Notice of Action designating dated December 22, 2009 (http://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=200809-0651-003#), designating the ICR as 0651-0063.
 USPTO (2009c). The response to comments contained in the revised Supporting Statement admits no error, makes numerous false statements, mischaracterizes comments to make them easier to rebut, and defends duplicative requirements based on a factor that is statutorily prohibited¾namely, saving the agency money. See 5 C.F.R. § 1320.5(d)(1)(iii): In the agency’s demonstration of ”practical utility,” the agency shall “seek to minimize the cost to itself of collecting, processing, and using the information, but shall not do so by means of shifting disproportionate costs or burdens onto the public” [emphasis added]).
 U.S. Patent and Trademark Office (2009b). This notice invites participation in a 3-hour public roundtable to be held in Alexandria, Virginia, on January 20, 2010. Although USPTO characterizes this effort as something akin to a do-over, the proposed changes are written as if they were amendments to the stayed final rule, which never went into effect. This approach makes effective public participation much more difficult than it should be.
 For more information, see the Environmental Protection Agency’s web site at http://www.epa.gov/nerl/research/nhexas/nhexas.htm. NHEXAS was a population-based (i.e., representative sample) pilot study of over 500 people in three areas of the U.S. to metals, pesticides, volatile organic compounds, and other toxic chemicals.
 Office of Management and Budget (1990).
 Office of Management and Budget (2006). Survey respondents and nonrespondents tend to be different. The magnitude of potential nonresponse bias rises nonlinearly as the response rate declines. The best way to overcome nonresponse bias is to prevent it¾hence the desire to use economic incentives to motivate participation.
 Belzer (2002, pp. 97-98): “The original supporting statement for NHEXAS implied that it would provide reliable estimates of the upper tail of an n–dimensional joint probability distribution of persons highly exposed across multiple chemicals and multiple pathways. Needless to say, the sample sizes necessary to obtain reliable estimates of this are much greater than the numbers that were proposed” (internal footnotes omitted).
 A new, and much more ambitious, human exposure study is getting underway under the auspices of the National Institute of Child and Human Development (NICHD). The National Children’s Study (NCS) proposes to collect an extraordinary amount of data from 100,000 children, beginning before conception and continuing until age 21. To date, NICHD has obtained only a generic OMB clearance authorizing very limited information collection for “Formative Research and Pilot Methodology Studies.” See Office of Management and Budget (2008). The NCS is not the kind of information collection in which either the sponsoring agency or OIRA should be lackadaisical about ensuring adherence to PRA principles and procedures. If active engagement does not begin very soon, the NCS will become mired in controversy and recrimination once NICHD begins actually trying to collect human data and PRA violations become frequent, severe, and harder to stop.
 See EPA (2009), OMB’s ICR record at http://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=200904-2070-001#section0_anchor, and Belzer (2009a).
 EPA’s purpose for imposing this mandatory information collection is to validate the very tests it is requiring test order recipients perform. This purpose is illegal under the PRA, which requires that agencies demonstrate actual practical utility before seeking information and does not permit agencies to mandate information collections that it hopes will validate themselves after the fact. If the PRA permitted agencies to rely on speculative practical utility, every practical utility claim would be sufficient.
 See OMB’s Notice of Action at http://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=200904-2070-001#.
 Id: “[Under the principles of the PRA, EPA should promote and encourage test order recipients to submit Other Scientifically Relevant Information (OSRI) in lieu of performing all or some of the Tier I assays, and EPA should accept OSRI as sufficient to satisfy the test orders to the greatest extent possible.”
 Census Bureau (2008).
 Cost is not clearly defined, and the term opportunity cost appears nowhere in the survey instructions. Cost and expenditure have different meanings, but PACE appears to treat them as synonyms subject to the varied interpretations of survey respondents.
 Census Bureau (2008, PDF p. 78). Section 131 authorizes the bidecadal census of “manufactures, of mineral industries, and of other businesses, including the distributive trades, service establishments, and transportation,” of which PACE is not a part. Sections 182 and 193 are similarly irrelevant. Section 224 places the penalty for nonresponse at no more than $500. However, Section 225 limits the imposition of penalties to “such inquiries as are within the scope of the schedules and questionnaires and of the type and character heretofore used in connection with the taking of complete censuses under subchapters I and II of chapter 5 of this title,” and “only after publication of a determination with reasons therefor certified by the Secretary, or by some other authorized officer or employee of the Department of Commerce or bureau or agency thereof with the approval of the Secretary, that the information called for is needed to aid or permit the efficient performance of essential governmental functions or services.”
 ATSDR was established by the Comprehensive Environmental Response, Compensation, and Liability Act of 1980 (CERCLA, or “Superfund”). Its mission was narrowly tailored to focus on estimating human health effects from Superfund sites. Since then, the Agency has successfully expanded its mission into matters far afield from human health effects resulting from exposure to hazardous waste sites.
 5 C.F.R. § 1320.3(l).
 See, e.g., Executive Order 12,866, § 1(b)(1) (“Each agency shall identify the problem that it intends to address…”).
 5 C.F.R. § 1320.7.
 This involves more than merely issuing memoranda to senior officials nominally responsible for PRA compliance reminding them of their responsibilities. OMB officials have done that in 2001 (Graham and Lefkowitz 2001), 2002 (Graham 2002a, 2002b), 2003 (Graham 2003), 2004 (Graham and Newstead 2004; Johnson III and Graham 2004), 2005 (Graham and Newstead 2005), 2005 (Graham and Newstead 2005), and 2006 (Arbuckle 2006). By now it should be apparent that this does not work.
 See 5 C.F.R. § 1320.5(a)(1)(i), referencing § 1320.8.
 See 5 C.F.R. §§ 1320.5(a)(1)(ii) and 1320.5(a)(1)(iii)(F).
 See 5 C.F.R. § 1320.5(a)(iii), referencing the certification requirements in § 1320.9 and the procedural requirements in §§ 1320.10-12.
 See 5 C.F.R. § 1320.5(b)(2)(ii).
 This could be a special case of the problem noted in subsection 2 above, or it could reflect the efforts of lower-level personnel to circumvent effective internal agency oversight.
 OMB’s quiet approval in December 2009 of bootleg paperwork burdens imposed by USPTO’s internal Board of Patent Appeals and Interferences rewards the Patent Office for its prior and longstanding violations of law. See footnotes 41-44 and the surrounding text.
 The checkbox certification looks like this:
On behalf of this Federal agency, I certify that the collection of information encompassed by this request complies with 5 CFR 1320.9 and the related provisions of 5 CFR 1320.8(b)(3).
The following is a summary of the topics, regarding the proposed collection of information, that the certification covers:
- o (a) It is necessary for the proper performance of agency functions;
- o (b) It avoids unnecessary duplication;
- o (c) It reduces burden on small entities;
- o (d) It uses plain, coherent, and unambiguous language that is understandable to respondents;
- o (e) Its implementation will be consistent and compatible with current reporting and recordkeeping practices;
- o (f) It indicates the retention periods for recordkeeping requirements;
- o (g) It informs respondents of the information called for under 5 CFR 1320.8 (b)(3) about:
(i) Why the information is being collected;
(ii) Use of information;
(iii) Burden estimate;
(iv) Nature of response (voluntary, required for a benefit, or mandatory);
(v) Nature and extent of confidentiality; and
(vi) Need to display currently valid OMB control number;
- o (h) It was developed by an office that has planned and allocated resources for the efficient and effective management and use of the information to be collected.
- o (i) It uses effective and efficient statistical survey methodology (if applicable); and
- o (j) It makes appropriate use of information technology.
If you are unable to certify compliance with any of these provisions, identify the item by leaving the box unchecked and explain the reason in the Supporting Statement.
 As an agency, the number of FTEs at OMB has not kept pace with the Office’s statutory and executive responsibilities. Successive OMB directors of both parties have tried to limit the growth of FTEs government-wide, and used OMB as a “principled example” of frugality. There is little evidence that this example has had much effect on the agencies OMB supervises. Shortly after it was created, OIRA comprised better than 10% of OMB’s staff. Successive OMB directors have used OIRA as a source of slots for use elsewhere. Today, OIRA comprises maybe 5% of OMB’s professional staff.
 I searched the GPO website for [“information collection request” (“60-day notice” OR “30-day notice”)].
 One hit was not an information collection request. One hit did not include the search terms. My list is not exhaustive because some 60- and 30-day notices escaped detection using my search logic. I am aware of one such example (DHHS Centers for Disease Control and Prevention, “Proposed Data Collections Submitted for Public Comment and Recommendations,” 74 FR 55559), which does not use any of my search terms. This underscores the difficulty the public has in learning about ICRs.
 A typical generic ICR title is “Agency Information Collection Activities: Proposed Collection; Comment Request.”
 Agencies issuing notices that consistently included useful subject matter information in the Federal Register title were the Environmental Protection Agency, the Department of Homeland Security, and the Department of the Interior. Their practices could easily be extended to the rest of the government.
 Although I have wide experience in federal regulation, I am unfamiliar with the subject matter in many of these notices. Thus, I do not know whether the texts in these notices are adequately descriptive. It is nevertheless my conviction that ICR notices have a long way to go before any reasonable person would characterize them as informative.
 See 5 C.F.R. § 1320.5(b)(2)(ii): “An agency shall provide the information described in paragraph (b)(2)(i) of this section in a manner that is reasonably calculated to inform the public.”
 This remedy would fail if OIRA did not make its actions public or if it failed to actually disapprove ICRs founded on defective notice. The public can be enlisted to help, but ultimately OIRA has to do its job.
 Office of Management and Budget (2009, pp. 55270-55271).
 This occurred during Executive Order 12,866 OMB review of the pair of draft final rules that became USPTO (2007a), subsequently estimated to include about $30 billion per year in paperwork burdens (Belzer 2008c). The text of the combined final rule was not publicly known. Nevertheless, members of the public presented to OMB credible information indicating that paperwork burdens were likely to be extraordinarily large. See Boundy (2007), Enclosure 3, Attachment M. OMB concluded review of the final rule without designating the rule as economically significant or requiring the agency to prepare a Regulatory Impact Analysis.
 Katznelson (2008) and Belzer (2008a, 2008b).
 See 5 C.F.R. § 1320.8(d)(1): “Before an agency submits a collection of information to OMB for approval, and except as provided in paragraphs (d)(3) and (d)(4) of this section, the agency shall provide 60-day notice in the Federal Register, and otherwise consult with members of the public and affected agencies concerning each proposed collection of information…”
 OIRA experienced precisely this kind of controversy when it approved EPA’s EDSP Tier 1 Test Order ICR subject to Terms of Clearance directing EPA not to impose mandatory testing where doing so violated the Paperwork Reduction Act. Rep. Edward J. Markey sent a letter to OMB Director Peter Orszag asking him to explain “the basis for OMB’s actions” but without any apparent recognition of its statutory basis. See Markey (2009) and the reply by Orszag (2009).
 Belzer (2009b, pp. 55-56).
 See 44 U.S.C. § 3512 and 5 C.F.R. § 1320.6.
 See 5 C.F.R. § 1320.6(d): “Whenever a member of the public is protected from imposition of a penalty under this section for failure to comply with a collection of information, such penalty may not be imposed by an agency directly, by an agency through judicial process, or by any other person through administrative or judicial process.”
 OIRA can, and should, demand that agencies be more forthcoming with respect to the paperwork requirements in proposed rules. There is no justification for keeping this information confidential, yet agencies routinely provide inadequate information about and justification for the paperwork burdens associated with proposed rules.
 Ironically, OMB promptly posts notices of meetings held at the request of outside parties related to regulations under review pursuant to Executive Order 12,866. See http://www.whitehouse.gov/omb/oira_default/. Whenever third parties provide written materials to OMB during an EO 12,866 review, these materials also are posted promptly. Given that PRA reviews are supposed to be public, this contrast in OMB’s disclosure practices is striking and hard to explain.
 To be concrete, there are four unique versions of the Supporting Statement for USPTO’s ICR 0651-0031 posted at Reginfo.Gov in Microsoft Word or PDF format. These versions have very significant differences about which the public has a right to know but which OIRA chooses to obscure from public view.
The most recent version (created April 22, 2008) is listed on the web site and can be downloaded directly (http://www.reginfo.gov/public/do/DownloadDocument?documentID=44055&version=4). Three previous versions are not visible to the public but can be downloaded by those who know the URLs:
- Versions 0 and 1: http://www.reginfo.gov/public/do/DownloadDocument?documentID=44055&version=1, created September 26, 2007;
- Version 2: http://www.reginfo.gov/public/do/DownloadDocument?documentID=44055&version=2, created January 4, 2008; and
- Version 3: http://www.reginfo.gov/public/do/DownloadDocument?documentID=44055&version=3, created April 11, 2008.
 Comments made by Gary Timm, “Status of the U.S. Endocrine Disruptor Screening Program (EDSP),” The Endocrine Disruptor Screening Program: What Can Screening Results Tell Us About Potential Adverse Endocrine Effects? International Society for Regulatory Toxicology and Pharmacology,” September 9, 2009.
 Belzer (2008d).
 44 U.S.C. §§ 3501(2) and (4).
 Belzer (2002, p. 98): “OMB acts like a protective father who does full FBI checks on all the gentlemen who come calling to court his daughters, but doesn’t bother to verify that the girls ever came home, much less before midnight with their honor intact.”
 See USPTO (2006c, p. 67). The 60-day PRA notice for the companion NPRM reported identical burdens (2006b, p. 58).
 In its recent Notice of Action approving EPA’s EDSP Tier 1 Test Order ICR, OIRA’s Terms of Clearance placed additional public participation requirements on EPA:
“OMB requests that EPA provide a report re-estimating the burden of this information collection based on responses to the Tier I test orders, including the use of cost-sharing and data compensation, the submission and acceptance of existing data and OSRI, and description of any instances in which submission of OSRI was deemed insufficient to satisfy the testing order. OMB requests this report prior to or at the time of submission of revision of this information collection to cover additional chemicals. In addition, in order to ensure that EPA has maximized the practical utility of the Tier I assays as the program moves forward, EPA should ensure sufficient opportunity prior to submission of any revision to this collection for public comment and peer review of the EPA tools to be developed to guide agency decisions on whether a chemical must proceed to Tier II, including the Weight of the Evidence Approach and Standard Evaluation Procedures (http://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=200904-2070-001).
OIRA’s historic willingness to tolerate substandard agency performance weakens the public’s confidence that it will actually require EPA to meet these conditions. Similarly, OIRA’s ambivalence to public participation¾ evident in OIRA’s peculiar disclosure practices¾necessarily creates doubt about whether OIRA will make these reports public or insist that the public be invited to comment on them.
 USPTO (2005a, [0651-AC12]; 2005b, [0651-AB93]; 2005c, [0651-AB94]; 2005d, [0651-AB95]; 2006d, [0651-AC00]).
 USPTO (2005b, 2005c).
 Tafas v. Dudas. 541 F. Supp. 2d 805 (E.D. Va.)(2008).
 5 C.F.R. § 1320.18(b).
 Belzer (2009b, cover letter at 2).