9. EVALUATING PROGRAM SUCCESS AND ESTABLISHING ACCOUNTABILITY

INTRODUCTION

Information can be a powerful and vital tool for successfully implementing an enforcement program. Information about program activities and results can ensure that individuals responsible for pursuing enforcement are, in fact, doing so consistently and fairly using established procedures and strategies. Information can help managers adjust enforcement programs to changing conditions and lessons learned as the program is implemented. Periodic program evaluations to gather information about program activities and results serve many purposes:

· Evaluating Program Strategy. Evaluation helps program managers determine whether the strategies they are using to achieve compliance are working. Results of evaluations are used as a basis for identifying problem areas and making changes to improve effectiveness.

· Internal Accountability. Periodic evaluations of success provide a basis for establishing a system to hold program personnel accountable for the implementation and effectiveness of the program. Establishing an accountability system involves defining performance goals and/or measures, obtaining commitments from program personnel to achieve those goals/measures, and evaluating their performance against those goals/measures. Where necessary, action is taken to improve performance. Accountability is valuable to ensure the quality of the program at all levels, from entry-level personnel to senior management.

· Creating Deterrence. Periodic reporting of program activities and successes to the regulated community contributes to deterrence by raising awareness that there is a good chance violations will be identified and responded to. Such reporting will be effective only if the program has been active and successful.

· Public Accountability. In some countries, enforcement programs may be required by law to report their progress and achievements to the public. Program evaluation provides the basis for public accountability. This accountability can be an important force in shaping program strategies and priorities. The U.S. enforcement program, for example, is continually scrutinized by the members of the U.S. Congress, who were elected by the public. Members of Congress may request hearings and reports to learn about program activities. Members of the public may contact their Congressional representatives at any time to express satisfaction or dissatisfaction with a program.

This chapter discusses issues in and approaches to evaluating program success.

ISSUES IN MEASURING SUCCESS

Measuring the success of a enforcement program is not easy. In the United States, there is a continuing debate about how success should be measured. Many parameters can be used to evaluate program effectiveness. Some measure results, such as improvement in environmental quality and rates of compliance. Some measure activity levels such as inspections and enforcement actions that contribute to deterrence. Others provide qualitative assessments of program performance and direction. Program measures include (see also Figure 9-1):

· Environmental results.

· Compliance rates.

· Progress in returning significant violators to compliance.

· Measures of compliance monitoring.

· Number of enforcement responses.

· Timeliness of enforcement responses.

· Monetary penalties assessed.

· Measures of technical assistance.

Each of these measures (discussed below) has advantages and disadvantages. Several measures must be used to gain a meaningful assessment of program effectiveness. Key questions to ask when considering which measures to use include:

· How accurate is the measure?

· What resources are needed to obtain the necessary data?

· How frequently should data be collected?

· Who will collect the data?

· How should the data be reported, and to whom?

· Who will analyze the data? What will they analyze for?

· Where will the data be stored?

· Will the data be computerized?

Collecting and processing reliable information on compliance and enforcement can be a constant challenge. For example, all personnel involved in gathering or analyzing data need to clearly understand exactly what data should be reported. Problems can arise if different individuals within a program have different interpretations of what data are needed.

Another challenge is that different levels of an enforcement program may have different data needs. Local personnel, for example, may prefer to focus their resources on data they consider valuable for evaluating program performance. Program personnel at a national level may have different priorities. National data systems will benefit if they are designed from the bottom up. Since local personnel collect the data, they will have a greater incentive to gather accurate data if they believe the data will be useful to them.

Mechanisms will be needed to gather and store the data, and to transfer it at appropriate intervals to other program levels that will analyze the data. A schedule for issuing reports of the analysis will also be needed. Policymakers may also wish to conduct special studies to analyze program strategy and success, and recommend improvements. These studies could examine issues such as:

· The effectiveness of various program policies, e.g., which promotional vehicles were successful in reaching the regulated community, the policy for identifying and screening violators.

· The effectiveness of various enforcement techniques.

Such analysis would be useful when reviewing and refining program priorities and strategies.

MEASURES OF SUCCESS

Success can be measured in two basic ways. One way involves setting goals or targets (for example: a certain number of inspections should be conducted each year), and then comparing actual activity to the goal. The second way involves tracking results, i.e., looking for trends and changes in activities or results over time (for example, a finding that 25% more inspections were conducted this year than last year may indicate an improvement in this activity). Tracking can either be routine (e.g., annually) or periodic. Tracking can be applied to any of the success measures described below. The goal-setting approach works only when realistic goals can be set; this is possible and appropriate for only some of these measures, as described below.

Environmental Results

Improved environmental quality is the ultimate goal of any environmental program and therefore is the most desirable measure of success. The types of environmental results that can be measured include overall environmental quality, reduction in pollutant releases, and risk reduction. Unfortunately, these measures have several shortcomings:

· There can be a significant lag time between the compliance promotion and/or enforcement response activity and the resulting improvement in environmental quality.

· It is hard to link changes in environmental quality to specific sources or specific compliance actions.

· Other factors, such as changing weather patterns or economic conditions, may affect environmental quality and therefore the accuracy of this measure.

· Compliance with some environmental requirements does not result in measurable improvements in environmental quality.

Compliance Rates

Compliance rates are one of the best overall measures of enforcement success. High compliance rates are the ultimate goal of most U.S. programs. Nevertheless, this measure also has shortcomings:

· Compliance rates rely on the thoroughness and frequency of inspections and/or on the accuracy of self-reported data. Compliance rates will not be reliable if these data are not thorough or accurate enough.

· A lower compliance rate may mean that the program is doing a good job of detecting violations, that the program is using stringent standards for compliance, and/or that the regulatory requirements are stringent.

· A high compliance rate can be misleading if the most significant pollution sources remain out of compliance, or if sources in compliance fail to stay in compliance.

Because of these shortcomings, U.S. programs find it difficult to hold managers accountable for improvements in compliance rates. U.S. programs do, however, use compliance rates to suggest specific areas requiring management attention.

If compliance rates are used as a measure of success, policymakers will need to agree on what constitutes compliance. For example:

· Does compliance mean achieving the required emission levels or meeting a schedule for compliance set forth in an enforcement agreement?

· Should the compliance rate cover any and all requirements, no matter how minor, or just the most significant requirements?

· How should repeat violations be reported? For example, how should sources be reported that are in compliance during the reporting period, but which are known to regularly go in and out of compliance?

· What influence should the percentage of sources of unknown status have on the evaluation of compliance rates? For example, if a particular compliance rate is shown for 10% of facilities for which data exist, what assumptions are made about the other 90%?

· What data gathering is needed to ensure that facilities that are in compliance continue to stay in compliance?

Progress in Returning Significant Violators to Compliance

Significant violators are those violators that have the greatest impact on environmental quality. Bringing them into compliance will therefore have the greatest immediate impact on environmental quality. It may also have an important deterrent effect, since significant violators are often relatively large and well known sources within the regulated communities. This indicator is appropriate for both tracking and goal-setting. It is important to remember that this indicator does not provide any measure of success achieved in that portion of the regulated community that are not defined as "significant violators."

The U.S. has used this measure since the late 1970s. It is one of that country's most successful management tools. At first, the U.S. program officials identified the most significant pollution sources throughout the nation and proceeded to take action against them. This effort brought many large industries into compliance. However, enforcement activity declined rapidly when this initial list was exhausted.

In 1985, the U.S. adopted a new system that does not single out particular industries. Policymakers developed national criteria for what constitutes a significant violator. They also defined what actions should be taken for particular types of violations. Program officials must identify significant violators in their jurisdiction, and make commitments to taking specific actions against a certain number of significant violators every 3 months. Sources are tracked until full compliance is achieved. Records are kept of the number of significant violators identified, the number and type of actions taken, and the results of those actions. Performance is evaluated based on how closely these goals are met. The lists of significant violators are made publicly available.

This approach has several advantages:

· It tracks not only actions taken, but results achieved. Actions and results can be easily associated.

· The system encourages actions that will have significant environmental benefits.

· Enforcement program managers can analyze the data for patterns of compliance across industry, companies, and environmental media.

· Publicizing the lists of significant violators may encourage other sources to achieve and maintain compliance.

Measures of Compliance Monitoring

Another measure of success, appropriate for both tracking and goal-setting, is how well an enforcement program monitors compliance. Several measures can track progress in this area:

· The number of inspections.

· The quality of inspections.

· The appropriateness of the targets of inspection.

· The quantity of self-reported data received.

· The quality of self-reported data received.

The number of inspections is probably the easiest of these indicators to track. This indicator provides a qualitative measure of program success in creating an enforcement presence.

The United States uses these indicators in its enforcement programs. Program officials set goals for and report on the number of inspections. Policymakers develop national criteria for effective inspection strategies, and program officials evaluate the strategies against these criteria. The United States also conducts oversight inspections to assess the quality of program inspections. Oversight inspections are conducted by program inspectors or consultants either separately or simultaneously with local inspectors.

One issue in measuring compliance monitoring is that well-targeted, high quality inspections will probably increase the number of violations detected and thus lower the compliance rate.

Number of Enforcement Responses

Legal action is the ultimate weapon in the arsenal of environmental enforcement tools. Measures of enforcement responses may therefore be of particular interest to members of the public and nongovernment organizations that are concerned about environmental quality. In the United States, for example, this measure is viewed by the public and by the U.S. lawmakers as an indication of program managers' commitment to gain compliance, and it is therefore closely tracked.

Despite its potential importance in public relations, this indicator has important disadvantages:

· The fact that an enforcement action has been initiated does not mean that compliance will be achieved in a timely and effective manner. The litigation process can result in lengthy delayed compliance schedules.

· Legal action is the most costly enforcement response. An emphasis on legal action may divert attention and resources from other important program activities essential to program success. This may be a particular concern if the regulated sources are small and numerous.

· The number of enforcement responses may depend, in part, on the degree of noncompliance. For example, it may be easier to bring successful enforcement action in the early stages of a program when there are many obvious violators, than at later stages when (if the program has been successful) violations are less dramatic and less obvious.

To use this indicator, policymakers must decide exactly what will be counted: total number of legal cases initiated; a breakdown of the types of cases by severity of violation, number of sites involved, multiple violations, or repeat violators; the number of cases won, etc. These indicators are not appropriate for goal-setting, because making program managers responsible for meeting quotas for enforcement response could undermine the objectivity of the program in evaluating whether or not sources are in compliance.

Timeliness of Enforcement Responses

One of the best indicators of a program's efficiency is the time it takes to either (1) respond to a violation, or (2) achieve compliance. Ideally, many types of enforcement responses should be as swift as possible so that the source can be returned to compliance as quickly as possible. Timeliness can be evaluated by monitoring trends and, sometimes, by comparing actual results against predetermined goals. For example, monitoring trends is particularly appropriate for measuring time to achieve compliance, since so many factors influence this result. Timeliness can also be measured by setting goals for different types of enforcement actions. Success is then measured by comparing the actual schedules with these timeliness goals. Goals can only be set for those types of enforcement actions that consistently take a predictable time to complete. These are usually the earlier and more routine enforcement actions. Enforcement actions involving later stages of legal procedures are generally too unpredictable to be evaluated in this way. Also, timely response may not be possible or appropriate in some cases, such as criminal cases, that required detailed investigation before an enforcement action is filed. Care may be necessary to ensure that use of timeliness as a measure of program success does not encourage enforcement personnel to take simple administrative action rather than pursuing a more time-consuming enforcement response.

Monetary Penalties Assessed

This indicator is simply the total number and/or value of penalties assessed as a result of enforcement actions. Trends in this indicator are used to measure success, since it is not possible or appropriate to set goals for how many penalties should be assessed during a particular time period or how severe the penalties should be. This indicator may not be a good means of holding managers accountable for successful enforcement activity because there is generally a significant lag time (sometimes years) between the initiation of an enforcement action and assessment of a monetary penalty.

In the United States, reports of the total value of monetary penalties assessed for environmental violations are prepared annually.

Measures of Technical Assistance

One measure of success for programs with an emphasis on compliance promotion is the extent and effectiveness of technical assistance provided by the program to the regulated community (see Chapter 5). This success measure is appropriate for both tracking and goal-setting. Several measures can track progress in this area:

· The number of facilities that have received technical assistance.

· The increased compliance achieved by facilities receiving technical assistance.

Other Measures

The search for useful measures of enforcement success is an ongoing and creative process. Other measures in addition to those described above may prove useful, such as the rate of recidivism (i.e., Do those subject to enforcement response maintain or improve compliance in the future?) and timeliness of the return to compliance once a violation has been detected (i.e., How quickly is compliance achieved?).