GAO: SSA Demonstration Projects Fail to Yield Reliable Information Due to Management Failures

A recent report by the Government Accountability Office (GAO) concluded that, as of August 2008, the SSA had not fully implemented recommendations the GAO made in 2004 and did not have written policies and procedures governing how it should review and operate its demonstration project program.

SSA did not conduct impact evaluations

Over the last decade, the SSA has initiated 14 projects under its demonstration authority to test both Disability Insurance (DI) and Supplemental Security Insurance (SSI) program changes--six related to DI, six related to SSI, and two examining both programs jointly.

Of these 14 projects, the SSA has completed four, cancelled five and had five projects in progress as of August 2008. In general, the agency has used its demonstration authority to conduct projects to consider changes to program administration and benefits counseling, as well as to test options for providing employment assistance and health benefits for beneficiaries. In total, the SSA had spent about $155 million on these demonstration projects as of April 2008, and officials anticipate spending another $220 million in the coming years on those projects currently in progress. Yet the SSA's demonstration projects have yielded limited information about the impact of the policies and programs being tested.

The GAO found that the SSA did not conduct impact evaluations --assessments of a project's effects compared to what would have happened in its absence--for two of its completed projects because the data it planned to use was not available. In addition, the agency intended to evaluate the impact of five other projects but could not do so to any great extent because significant challenges led it to cancel the projects in the early stages, and thus no data was available to assess the policies and programs being tested.

The SSA was able to obtain some preliminary information from one of its cancelled projects and also obtained some information from the two completed projects it evaluated, although the evaluations' findings had limitations. In one case, the agency could not separate the effects of its project from the effects of a similar federal project that was also under way. It has also obtained some preliminary information from one of the five projects currently under way, and agency officials believe that these projects will ultimately yield useful information. However, it is too early to tell whether the information obtained will be used to perform changes to disability program policy. The projects are expected to continue until 2010 or later.

Written management controls still lacking

Although the SSA has taken steps to improve its demonstration projects, the agency continues to lack written management controls to ensure that the projects yield reliable information about their impacts. For 11 of the projects initiated, the SSA has used experimental or quasi-experimental designs--methods commonly used by research professionals for demonstration projects--that the GAO determined were strong or reasonable when assessed against professional research standards. The agency has also used external research professionals to assist with the design, implementation or evaluation of 12 of the 14 projects, and appointed new management in 2007 to oversee the demonstration project program. The new management team subsequently conducted an internal review of the 10 demonstration projects that were underway at the time it was created and cancelled five projects that were unlikely to yield reliable information because they faced significant limitations or challenges or would have been duplicative of other ongoing research.

Specifically, the SSA still does not have written policies and procedures for its managers and project officers to follow as they design, implement and evaluate its demonstration projects. Absent such protocols, the agency sometimes did not apply standard research practices, such as conducting pilot phases or obtaining sufficient stakeholder input, which led to data limitations and project cancellations.

Written reporting system necessary

The GAO recommendations continues to recommend establishing written policies, procedures and mechanisms for managing and operating the demonstration projects. It states that these policies and procedures should be consistent with standard research practices and internal control standards in the federal government. In response to the GAO draft report, the SSA generally agreed with the recommendation and the need to develop a guidebook to assist its staff in the design, implementation and evaluation phases of its demonstration projects. It considers its current guidebook a work in progress. The SSA also stated that the agency has taken steps in recent years to address the GAO's prior recommendations. However, the GAO maintains that additional steps are needed to fully implement them.

Reprinted with permission. © CCH

GAO SSA Demonstration Projects Fail to Yield Reliable Information Due to Management Failures A recent report by the Government Accountability Office (GAO) concluded that, as of August 2008, the SSA had not fully implemented recommendations the GAO made in 2004