Compliance Assurance Program Evaluation Principles

INTRODUCTION

These principles have been developed jointly by the States and EPA to help guide
compliance assurance program evaluations. The principles will be used by EPA to evaluate
the compliance and enforcement components of environmental programs that have been
delegated to States. The principles define the elements of a successful compliance assurance
program and formulate the basic operating principles for conducting an evaluation.

These principles represent the first phase of a two-phase effort to develop principles to guide
program evaluations. This document constitutes Phase One and pertains specifically to
focused evaluations of compliance assurance programs. Phase Two will be undertaken in
the near future to develop principles to guide EPA reviews of a State's overall program,
referred to as a "contextual" review. The contextual review will address the full range of a
State program's activities and accomplishments and will include compliance assurance as
one component of the overall program.

BACKGROUND

Compliance assurance program evaluations have traditionally taken a narrow view of the
tools available for achieving compliance. These principles move beyond evaluating
programs in terms of traditional enforcement activity measures, by taking a broader view of
all the tools which can be used to achieve compliance and by focusing on measuring the
environmental results of compliance assurance activities.

These principles strive to clarify our expectations of delegated compliance assurance
programs and what is being measured as part of the compliance assurance program
evaluations, because the evaluation process and criteria have not always been clear in the
past. While the principles define the basic elements of a successful compliance and
enforcement program, each EPA program will need to develop specific measures, based on
its unique objectives and requirements, to evaluate how well the State program is performing
within each of these elements.

These principles are designed to be consistent with and complement other planning and
evaluation processes, including the PPA process, State and EPA strategic planning efforts, and
compliance assurance agreements. The principles will continue to be refined over time so
that they are as useful as possible to EPA and the States in conducting focused compliance
assurance program evaluations, and will be expanded to also serve as principles for
contextual program reviews.

I. PROGRAM EVALUATION GOALS AND OBJECTIVES

The primary goal of conducting program evaluations is to ensure a high level of
program performance through the use of a variety of compliance assistance and
enforcement tools which results in high rates of compliance, provides credible
deterrence, and achieves positive environmental results.

The specific objectives for conducting program evaluations are as follows.
The initial program evaluations will help establish a compliance program profile
which will be used as a starting point or baseline, as appropriate, against which
progress towards mutually agree-upon goals can be measured over time. This profile
is important to help clearly and objectively identify both progress and problem areas.

II. FREQUENCY OF EVALUATIONS

Formal Evaluations
EPA will align evaluations with the PPA/ PPG process by conducting program
evaluations on a mutually agreeable rotating cycle that is identified in the PPA. At a
minimum, each program will be evaluated once during a six year period. The
evaluation may be comprised of a single evaluation of the compliance assurance
program, or of several smaller, focused evaluations of one or more individual
evaluation areas (as identified in Section III of this document) conducted at different
times during the six year period.
Routine/ Ongoing Interaction
Ongoing dialogue regarding program activities should continue in the interim
between formal evaluations through a variety of routine communication mechanisms
such as conference calls and meetings at both staff and management levels. States
and EPA will also track selected output/ outcome measures and will conduct oversight
inspections as defined in the PPA and compliance assurance agreements.

III. EVALUATION AREAS

Program Performance and Effectiveness

Achieving environmental results is the best indicator of a quality program. EPA and
State Agencies increasingly are striving to measure program performance in terms of
outcomes and environmental results (e. g., changes in emissions or in ambient air
quality). Traditionally, program performance has been measured in terms of agency
activities (e. g., number of inspections). At this point in time, EPA and State Agencies
will evaluate available information on environmental results-related measures but will
also continue to track activity-related measures. Over time it is expected that it may
be feasible to place greater emphasis on environmental results measures as they are
improved, and that activity measures may be de-emphasized.
EPA and State Agencies will analyze, and present available information on, each of
the following types of measures:

Complete, Accurate, and Current Knowledge of the Regulated Community
A successful program is based upon an inventory of regulated sources, especially
priority segments, which is accurate, current, and as complete as practicable. The
data should in turn be accessible in an automated data system which is accurate and
up-to-date.
Appropriate Targeting, Inspection, and Monitoring Strategy
The targeting, inspection, and monitoring strategy should include the rationale and
approach for selecting sectors, facilities and/ or geographic locations on which to focus
compliance assurance efforts. The inventory of the regulated community, as well as
the national, regional, and state-specific priorities identified in the PPA, including
pollution prevention, should be utilized as a basis for targeting, inspecting, and
monitoring. The strategy should include field inspections and file reviews;
inspections are needed to establish a field presence that provides a deterrent effect,
encourages pollution prevention, and allows agencies to determine actual
compliance.
Balanced Use of Tools
It is important to have a dynamic compliance strategy which balances all the relevant
tools which will be used to achieve compliance. Each State should have a deliberate
approach that identifies the tools to be applied, in what balance, and how their
success would be evaluated. Available compliance tools include compliance
assistance and incentive programs, enforcement, inspections, compliance monitoring,
field presence/ accessibility to sources, publicity, and others. To evaluate the balanced
use of tools, EPA would: 1) examine the State's rationale for the balance of tools it is
employing; 2) consider how the strategy was applied during the period of review and
whether the intended focus was carried out; and 3) review how well the tools had
been utilized and/ or how effective they had been in achieving improved compliance
and environmental results. The EPA/ State strategy on compliance assistance, currently
under development, will also provide information on evaluating results of compliance
assistance.
Timely and Appropriate Response to Significant Violations
To serve as a general deterrent among the regulated community, a program must
provide timely and appropriate responses to significant violations. This includes
identifying high priority violators and aggressively pursuing these sources through
formal enforcement actions; recovering economic benefit; following through in a
timely fashion with adequate referrals for enforcement response; having no
unexplained delays or backlogs in enforcement cases; providing a rapid escalation of
response where noncompliance continues; adequately documenting violations and
substantiating penalties; and dispensing equitable, fair, and consistent treatment.
Accurate Recordkeeping and Reporting
A quality program maintains accurate and up-to-date files and records on source
performance and Agency responses that are reviewable and accessible.
Recordkeeping should include some documented rationale for penalties sought to
support defensibility in court, enhance negotiating posture, and lead to greater
consistency. A quality program reports relevant data that is accurate, reliable, and
complete to national data systems.
Clear and Enforceable Requirements
Requirements established through both formal and non-formal enforcement actions
should clearly define what a specific source must do by a date certain, in enforceable
terms, to ensure the State can effectively monitor and measure compliance changes
and as a basis for escalating its response to noncompliance.
Productive Relationship between Government and Regulated Community
Developing a productive relationship with the regulated community, while
maintaining an adequate deterrent effect, is important to achieving and maintaining
compliance. Evaluation of this relationship should include interacting with a
significant percent of the regulated community; using a variety of avenues of
communication to reach the regulated community; offering compliance assistance to
inform the regulated community about regulatory requirements; responding to
feedback from regulated community and the public; and being viewed by the
regulated community and the public as firm, fair, and professional.
Successful Partnerships with Co-Regulators
EPA and States, as co-regulators, work together to solve environmental problems. The
PPA and compliance assurance principles, in particular, are documents which reflect
this partnership through establishing productive ways of working together and
identifying mutually agreed upon priorities and commitments. This element should
be evaluated based on whether the State and EPA are each fulfilling the collaborative
spirit of the partnership and adhering to the specific principles established in these
documents.
Sound Program Management
A quality program should have internal management systems in place for self-evaluation,
monitoring program performance, reprioritizing efforts as necessary, and
addressing emerging issues. Program priorities should be well articulated throughout
the organization, and there should be a clear scheme for how the operations of other
agencies and levels of government fit into the program and its priorities. A quality
program should also ensure qualified and trained staff and mechanisms for helping
to ensure consistent application of the regulations, enforcement, interpretations,
equity, and other compliance issues.

IV. INFORMATION SOURCES

States have the option of providing a self-assessment at the initiation of the evaluation,
based on the evaluation criteria, to supplement other information sources listed here.
A variety of other data sources may be used, including PPAs, compliance assurance
agreements, significant violator tracking documents, State strategic plans, national data
systems, documents kept on-site, results of special initiatives, interviews with agency
staff or members of the regulated or affected community, relevant State laws and
policies, and results of oversight inspections.

V. COMMUNICATION

Communication should be open and constructive, and lead to productive resolution
of issues.

EPA Communication with the State
- Communication will be focused on the evaluation areas in Section III, and the
  commitments in the PPA and compliance assurance agreements.
- EPA will communicate to the State compliance assurance program strengths as
  well as opportunities for improvement, including:
State Communication with EPA
States are encouraged to provide feedback to EPA on the impact of EPA's direct
implementation and workshare efforts and on EPA's adherence to the State/ EPA
Compliance Assurance Principles, PPA provisions, and any other partnership efforts.

VI. PROCESS MANAGEMENT

Advance Work to Help Ensure a Successful Evaluation Process
- The evaluation scope, evaluation areas that will be focused on, measures,
  information sources, relevant policy documents, and resources will be clearly
  defined at least sixty days before the evaluation begins.
- While the evaluation areas and corresponding measures should be flexible
  (e. g., can include special initiative results), the areas and measures should be
  agreed to in advance so that there are "no surprises" (e. g., conducting
  compliance assistance special initiatives instead of core enforcement
  requirements).
Evaluation Feedback Steps
- EPA will discuss its findings with the State at an out-briefing within 30 days of
  the on-site review, prior to the issuance of the draft report
- EPA will provide the draft written evaluation to the State within 90 days after
  the evaluation has been completed
- The State has up to 30 days to provide a response back to EPA
- EPA will issue a draft final report within 30 days that reflects State's comments
- The State has up to 30 days to provide comments on the draft final report
- EPA will issue the final evaluation report within 30 days of receiving State
  comments. State views, if the State so requests, will be included in the final
  written evaluation.
Relationship to PPA and Compliance Assurance Agreements
- Program evaluations will occur within the framework of the PPA process:
  - The PPA will address whether (and in what areas) a program evaluation
    will occur in each State over the coming two years; and
  - States will be evaluated on the commitments identified in the PPA and
    Compliance Assurance Agreements and based on the evaluation areas
    in Section III of this document.
Evaluation Phases
These focused program evaluations can be conceived of as having three phases:
collection of information, interpretation of information, and the response delivered
regarding the evaluation.
Manager/ Staff Involvement
- Different people may be involved in each evaluation phase. Senior manager
  involvement is critical to the success of the evaluation and there will be a
  senior manager assigned to lead the evaluation. Senior manager involvement
  is most important during the process development and feedback phase.
- Appropriate EPA and State personnel involved in the evaluation will meet
  together as necessary during the evaluation to assure consistent understanding
  of issues and feedback.
- EPA will provide training on these principles to staff conducting the evaluation.
Improving the Process Over Time
- The frequency, scope and level of detail of the evaluation may be adjusted
  based on scope, detail and results of prior evaluations and other performance
  data (e. g., compliance rates). The PPA will reflect these adjustments over time
  in the evaluations.
- EPA/ States should agree on the process for assessing the quality of the
  evaluation performed.
- These compliance assurance program evaluation principles will be revised and
  expanded to serve as principles for contextual program reviews (i. e., review
  of the full range of program activities and accomplishments, with compliance
  assurance as a component of the overall program).
Differences Across States
- While States should not be held to different standards in the evaluations, it is
  important to recognize differences in size and composition of the regulated
  universe in each State and alternative methods States may use in addressing
  the evaluation areas identified in this document.
- EPA will use differential oversight to assess state performance and identify
  actions needed to correct problems or recognize good performance.

Integration with Existing Agreements

These compliance assurance principles reflect the current positions of the State Agencies and
EPA Region 10. As appropriate they can be used to assist in the implementation of existing
agreements. EPA Region 10 and the State Agencies will adhere to these principles when
developing any future state/ EPA agreements addressing compliance assurance program
evaluations.

Dispute Resolution

EPA and the State Agencies recognize that disputes may occur. All parties will attempt to
resolve these disputes promptly and at the lowest level. If disputes cannot be resolved within
seven days, they will be referred to the supervisor level. This supervisor referral and
resolution process will continue, if necessary, to the level of State Director and EPA Regional
Administrator.

Disclaimer

Nothing in these principles shall be construed to constitute a valid defense by regulated
parties in violation of any state or federal environmental statute, regulation, or permit. This
agreement is not intended to, and does not, waive any authorities available to the states and
EPA. Nor can this agreement be used to create a cause of action not otherwise available
against the states or EPA.

Compliance Assurance Program Evaluation Principles

Local Navigation