DRAFT
IALC Custom Impact
Evaluation Plan
2013-2014 Impact Evaluation
Submitted to:
Energy Division
California Public Utilities Commission
505 Van Ness Ave.
San Francisco, CA 94102
Submitted by:
Itron, Inc.
1111 Broadway, Suite 1800
Oakland, CA 94607
(510) 844-2800
July 2014
2013-2014 Custom Impact Evaluation Research Plan – Final Draft
Table of Contents
1 Introduction 1-1
1.1 Claimed Energy Impacts by IOU for 2013 Custom Projects 1-2
1.2 Goals and Objectives 1-3
1.3 Evaluation Priorities and Researchable Issues 1-4
1.4 Evaluation Overview 1-5
1.5 Recent Related Research 1-6
1.6 Evaluator Contact Information 1-6
2 Overview of IALC Custom Work Orders 2-8
2.1 Composition of 2013 and 2014 Claims 2-8
2.2 Work Order Claims Composition by IOU 2-9
3 Data Sources 3-1
3.1 Program Tracking Data and IOU Data 3-2
3.2 Project Specific Documentation 3-2
3.3 End Use Metering of Selected Projects 3-3
3.4 On-site EM&V Audits 3-4
3.5 Telephone Surveys of Participating Customers, Trade Allies, and Account Representatives 3-4
3.5.1 Decision-Maker Survey 3-4
3.5.2 Trade Ally Interviews 3-4
3.5.3 Account Representatives 3-5
3.6 Vendor Telephone Surveys 3-5
4 Coordination and Communication 4-1
4.1 Progress Reporting to CPUC 4-1
4.2 CPUC Approval of Work 4-1
4.3 Information Flow and Mechanisms 4-1
4.4 Coordination with IOUs 4-3
5 Project Practices Assessments 5-1
6 Communication and Feedback to IOUs 6-1
6.1 Early Results Reporting 6-1
6.2 IOU Input on Evaluation Methods 6-2
6.3 IOU Input on Evaluation Reporting 6-2
7 Sample Design 7-1
8 Impact Methods 8-1
8.1 Objectives and Goals Guiding the Impact Methodologies 8-2
8.2 Baseline Selection 8-2
8.2.1 Dual Baseline 8-4
8.3 Gross Energy and Demand Impact Evaluation 8-4
8.3.1 M&V Approach Overview 8-4
8.3.2 Rigor Levels 8-6
8.3.3 Site-Specific M&V Plan (Enhanced and Basic Rigor Levels) 8-6
8.3.4 EUL/RUL 8-6
8.3.5 Measure Cost 8-7
8.3.6 Reliability, Bias and Uncertainty 8-7
8.3.7 Energy Savings Methodology (kWh and therm) 8-9
8.3.8 Interactions Between Measures 8-10
8.3.9 Aggregation of Results 8-10
8.4 Project Practices Assessments 8-10
8.5 Net Energy and Demand Impact Evaluation 8-10
8.6 Impact Evaluation Procedures 8-10
9 Timeline 9-1
10 Work Plan and Budget 10-1
10.1 Overview of Original Studies from EM&V Roadmap Inventory 10-1
10.2 Proposed Changes 10-1
10.2.1 Original Budgets from EM&V Roadmap Inventory 10-2
10.3 Budget 10-3
List of Tables
Table 11: Claimed Energy Savings by IOU for 2013 Projects in the Custom Impact Work Order 1-3
Table 12: Evaluator Contact Information and Roles 1-7
Table 21: Work Order Claims Breakdown 2-9
Table 22: IALC Custom Work Order Claims by IOU 2-9
Table 23: NRNC Work Order Claims by IOU 2-10
Table 61: Schedule of Feedback and Reports to IOUs 6-2
Table 71: Summary of Overall Sample Sizes for IALC Impact-Related Effort 7-3
Table 72: Summary of IALC Population and Impact Sample 7-5
Table 91: Timeline 9-2
Table 101: Original EM&V Plan Studies and Budgets 10-2
Table 102: Proposed Work Orders and Budgets 10-2
Table 103: IALC Custom Budget by Task 10-3
Itron, Inc. ii Table of Contents
2013-2014 Custom Impact Evaluation Research Plan – Final Draft
1Introduction
This research plan addresses the impact evaluation of 2013-2014 California investor-owned utilities’ (IOU) energy efficiency programs, focusing on custom measures.
Impact evaluations serve many purposes including independent estimation of program and measure impacts, provision of recommendations to improve programs, support for cost-effectiveness analyses, development of data and findings in support of future program planning, and support for strategic planning. This evaluation plan has been developed to address the highest priority evaluation objectives for the custom projects under California Public Utilities Commission (CPUC) direction for the Industrial, Agricultural, and Large Commercial (IALC) programs. The scope of work includes independent estimation of gross and net savings, and development of findings and recommendations that can be used to improve program and measure effectiveness.
The measures and projects within the custom group are extremely diverse and site-specific, with savings that range widely across projects (by several orders of magnitude). As a result, an efficient evaluation approach requires extensive site-specific evaluation utilizing stratification by project size as the primary sampling approach, with possible segmentation across domains such as investor-owned utility (IOU), sector, program grouping, fuel, measure grouping, or end use. In addition, there are a very large number of programs with custom measures in the nonresidential sector. As addressed in Section 7 of this plan, there are a number of tradeoffs that must be considered in developing a sampling and reporting plan for this custom-focused work order.
Programs, measures, and measure groupings that fall within the custom measure impact evaluation are assessed in order to estimate gross and net energy savings. Deemed measures targeted for study in the Efficiency Savings and Performance Incentive[1] (ESPI) decision and P4 targeted measures with uncertain parameters will not be addressed in this Research Plan, as deemed measures will be mapped to other roadmap studies. Custom measures typically include heating and cooling, refrigeration, process, and miscellaneous end uses across the commercial, industrial and agricultural sectors. Project Practices Assessments (PPAs) will examine custom project impact estimation methods and procedures, and facilitate an assessment of IOU ex-ante performance for custom projects, in support of ESPI. These reviews feature assessments of project compliance with ex-ante review guidance and requirements, and conformance with policy guidance, with an emphasis on ex-ante gross savings development and methods. PPAs will be more focused assessments than Low Rigor Assessments (LRAs) included in WO033 with enhanced scoring that will facilitate further analysis.
This Research Plan describes the goals and objectives of the IALC Custom Impact evaluation; outlines the researchable issues; describes the custom claims to be evaluated; identifies data sources that will be used; presents the approach for sampling; summarizes the methods that will be used to determine gross and net impact; explains how coordination and communication will occur with other related evaluation studies; and discusses the need for and approach to providing timely feedback to the IOUs and the CPUC.
Energy savings claims from the measures assigned to the Custom impact work order represent a significant contribution to the overall savings portfolio for the IOUs’ energy efficiency programs (about 20% of electric kWh savings claims and 71% of gas savings claims in 2013 on a statewide basis). It is important to conduct an impact evaluation on the custom measures, not only due to their share of the IOU savings portfolio, but also because of the heterogeneous nature of these measures and the potentially high uncertainty of savings for custom measures. For 2013, the IOU tracking data for the program portfolio has thousands of entries statewide with statewide savings claims totaling 2,443 GWh and 408 MW. Statewide gas savings claims from measures total 56 million therms. While the evaluation will cover both 2013 and 2014 impact claims, the data referenced in this Research Plan describes only the 2013 time period, as that reflects the data available at the date of this plan. The 2014 summary data along with detailed sampling plans will be provided later in addenda as the data become available.
1.1 Claimed Energy Impacts by IOU for 2013 Custom Projects
Table 11 illustrates the claimed energy savings included in the custom evaluation using the current measure group mapping[2] to impact evaluation work order.
Table 11: Claimed Energy Savings by IOU for 2013 Projects in the Custom Impact Work Order
Energy Savings Claims by IOUIOU / Electric Energy (GWh) / Electric Demand (MW) / Gas Energy (Million Therms)
IALC Savings Claim
PG&E / 233 / 37 / 21
SCE / 215 / 34 / -
SDG&E / 50 / 6 / 0.8
SCG / 0 / 0 / 13
Total / 498 / 77 / 35
Portfolio Savings Claim
PG&E / 1,073 / 180 / 31
SCE / 1,146 / 190 / -
SDG&E / 215 / 33 / 0.4
SCG / 9 / 6 / 25
Total / 2,443 / 408 / 56
IALC Percentage of Claim
PG&E / 22% / 20% / 68%
SCE / 19% / 18% / -
SDG&E / 23% / 17% / 200%
SCG / 0% / 0% / 53%
Total / 20% / 19% / 63%
Note: SDG&E portfolio gas savings claims are less than the IALC savings claims because the portfolio savings include a substantial component of negative savings from lighting interactive effects that do not generally impact IALC claims.
1.2 Goals and Objectives
The overarching goals of this impact evaluation for custom measures and programs are to verify and validate the energy efficiency savings claims reported from IOU energy efficiency programs; to provide feedback on how well program procedures and savings methods comport with energy efficiency policies, requirements, and expectations; and provide recommendations on how custom programs can be further improved or refined to support overall and custom-related energy efficiency goals and objectives. This is accomplished by balancing cost, accuracy and precision in the data collection and analysis conducted. Gross savings, free ridership and net savings (kWh, kW and therms) will be estimated. Evaluation savings estimates span the full lifecycle of the relevant measures. IOU savings claims will be adjusted using evaluation-based gross realization rate[3] and NTG ratio estimates. A goal of this impact evaluation is to provide the IOUs with feedback that can be used to make any necessary corrections to improve their current programs, as well as feedback on what aspects of program design and implementation are successful. For this to be successful, evaluations must be timely in their reporting of results and feedback to allow utility staff adequate time for making necessary improvements. Impact reports will be prepared for each of the 2013 and 2014 claim periods, supplemented by periodic ad hoc feedback that will be provided to the IOUs in a memorandum format. These impact results and feedback will be designed to contribute to program and portfolio planning and policy-making for the next program cycle.
In order to extract the greatest value from this custom measure impact evaluation, the data and information collected will be leveraged to the greatest extent possible. Energy efficiency impact realization rates, baseline and standard practice assessments, program feedback, and measure cost data will be shared across research areas to support cost effectiveness analysis, current cycle program improvements, strategic planning, future program planning and design, and updates to the Database for Energy Efficiency Resources (DEER).
1.3 Evaluation Priorities and Researchable Issues
Estimation of gross and net savings (kWh, kW, therms) from IOU custom measures, programs and measure groupings is a high-priority research issue and the focus of the evaluation effort. The gross impact estimates for sampled projects will be based on field inspections, measurements, and extensive engineering analysis (i.e., M&V). The sampling domains for which impact results will be developed are discussed in Section 7, along with related issues such as number of sample points and estimation methods.
The priorities for this evaluation effort and the researchable issues that this evaluation seeks to examine are described as follows:
1. Estimating the level of achieved gross impact savings, determining what factors characterize gross realization rates, and, as necessary, assessing how realization rates can be improved.
2. Estimating the level of free ridership, determining the factors that characterize free ridership, and, as necessary, providing recommendations on how free ridership might be reduced.
3. Providing timely feedback to IOUs to facilitate program design improvements.
4. Determining whether the impact estimation methods, inputs, and procedures used by the IOUs and implementers are consistent with the CPUC’s policy and decisions and best practices.[4] The evaluation will identify issues with respect to impact methods, inputs, and procedures and make recommendations to improve IOU savings estimates and realization rates.
5. Improving baseline specification, including collecting and reporting on dual baseline. Estimating the extent of any program-induced acceleration of replacement of existing equipment and, in such cases, the RUL of the pre-existing equipment.
6. Collecting data and information to assist with other research or study areas, which could include measure cost estimation, cost effectiveness, updates to DEER, strategic planning, and future program planning.
1.4 Evaluation Overview
To the extent custom measures, measure groupings and programs share core similarities, the proposed evaluation approaches will be consistently applied. However, given the heterogeneity of custom measures, specialized and unique approaches by project are also expected.
This custom impact evaluation study will be guided by the California Energy Efficiency Evaluation Protocols: Technical, Methodological, and Reporting Requirements for Evaluation Professionals[5]. The following protocols will be used, along with other protocols developed or prescribed by CPUC:
n Impact Evaluation Protocol
─ Gross Energy Impact Protocol
─ Gross Demand Impact Protocol
─ Participant Net Impact Protocol
─ Indirect Impact Evaluation Protocol
n Measurement and Verification Protocol
n Sampling and Uncertainty Protocol
n Evaluation Reporting Protocol
The custom measure gross impact assessment will involve standard M&V approaches to the extent appropriate and practical, including on-site data collection, monitoring and analysis for a representative sample of custom projects. The gross impact analysis will: a) develop ex-post estimates of the energy and demand savings for each project in the sample, and b) apply those findings back against the full participant population to obtain population estimates of program impacts. M&V rigor levels will vary based on the size, complexity, and magnitude of savings. The evaluation team will utilize IOU and implementer collected site specific information, including M&V data, supplemented by data collection through this evaluation.
In addition, a net-to-gross assessment is planned using telephone survey data collection and self-report methods to derive net program impacts. Similar to using varying M&V rigor levels, net-to-gross rigor will be based on the size, complexity, and magnitude of savings. More information regarding the net-to-gross assessment is found in section 8 of this plan.
The proposed sample sizes for the overall 2013-2014 custom evaluation are discussed in Section 7, a brief summary of which is provided here. A sample size of 315 gross impact sample points is currently planned, consisting of: 88 large projects where full M&V analyses are planned and 227 small projects where a less intensive project analysis and verification is planned. Gross realization rates will be calculated for both project types. To the extent feasible the NTG points will be nested within the M&V gross impact sample.