Evaluation plans describe how to measure the impacts of your program and the effectiveness of your process for delivering residential energy efficiency products and services. The Evaluation & Data Collection component provides detailed information about the purpose and content of different types of evaluation plans. The Evaluation & Data Collection component describes two types of evaluations that are relevant to this handbook:
- Impact evaluations are evaluations of program-specific changes (e.g., changes in energy and/or demand use), directly or indirectly induced, associated with an energy efficiency program.
- Process evaluations are systematic assessments of an energy efficiency program for the purpose of documenting program operations at the time of the examination, and identifying and recommending improvements to increase the program’s efficiency or effectiveness while maintaining high levels of participant satisfaction.
This handbook focuses on developing program design and customer experience metrics and measurement strategies that will be useful for impact and process evaluations. They are:
- Program metrics, which measure the extent to which your program is achieving its goals and objectives. Program metrics will be primarily useful for impact evaluations (i.e., what impact are we having?), but tracking progress toward goals may also be useful for process evaluation (i.e., how can we enhance our impact?)
- Customer experience metrics, which measure how well you have integrated all of your program components to provide customers with a satisfying experience and run an efficient and effective operation. Customer experience metrics will be primarily useful for process evaluations.
Measuring the effectiveness of your program design and the quality of your customer’s experience involves:
- Metrics, which are quantitative expressions of program outcomes or activities (e.g., number of homes upgraded, percent energy savings).
- Measurement strategies, which are the actions taken to collect data for your metrics. These strategies describe what will be collected, who will collect it, and with what frequency.
- A systematic process for information sharing and review to periodically examine the metrics and assess your program delivery process and customer satisfaction.
Handbooks for Marketing and Outreach, Financing, and Contractor Engagement & Workforce Development all describe a similar process of developing metrics, measuring them, and developing a plan for sharing and reviewing data to evaluate each of these specific program components.
Your approach to measuring the effectiveness of your program will be partly based on the amount of funds allocated to evaluation in your program’s budget. Many programs have found evaluation to be an essential program management function. In addition to providing insights to help program managers make informed decisions, evaluation data may also be necessary to demonstrate to funders or regulators that you are achieving your goals.
Key steps are:
- Establish program metrics that reflect your goals and objectives
- Establish direct and indirect customer experience metrics
- Document measurement strategies for collecting data
- Establish responsibilities for internal review of evaluation results.
Establish program metrics that reflect your goals and objectives
To develop metrics, start with your program goals and objectives. Each of your program objectives should link to an overall program goal and include a specific measurable target and timeline. Metrics will help you measure whether you are meeting your objectives. The table below illustrates the relationship between objectives and metrics.
Source: U.S. Department of Energy, 2014.
Metrics to assess progress toward your program goals may include:
- Activity metrics. These include the number of assessments, upgrades, loans, rebates, or other program activities.
- Participation metrics. These include the number or percent of homes in an area that are participating in the program, have been assessed, have been upgraded, and have achieved other program milestones. Participation metrics also include conversion rates, which measure how many homeowners are progressing through all steps in the upgrade process (e.g., the percentage of customers that have undertaken an upgrade, out of the total number receiving an assessment).
- Energy and demand savings attributed to the program. Energy and demand savings can be estimated using measured energy consumption before and after upgrades are completed (e.g., billing analysis), through simulation modeling, or deemed savings estimates. Savings targets can be expressed on an annual or lifetime basis.
- Cost savings for customers. Cost savings are typically estimated based on calculations of energy savings and energy prices.
- Other outcomes, such as gallons of water conserved, greenhouse gas emissions reduced, etc. Measurement of these outcomes is often based on estimates.
Benchmarking for Success
To effectively assess your program’s progress toward your goals and objectives, it is important to measure progress against clear benchmarks. The U.S. Department of Energy’s “Guide for Benchmarking Residential Energy Efficiency Program Progress with Examples” can help you identify metrics to measure goals, collect information on progress, and analyze and share results. Learn how programs have used metrics to effectively engage customers and achieve savings, and how to use a step-by-step action plan to develop your program’s benchmarking strategy. The guide provides 26 recommended benchmarking impact indicators that you can use to assess your total impact. By using metrics to visibly identify program performance relative to your goals, you can better enable yourself to make adjustments and improve your program design.
Calculating Energy Savings
Energy savings are one of the most critical program metrics to be tracked, reported, and ultimately evaluated. The methods you use to estimate savings resulting from your program will depend on your program design, measures promoted, and—for some programs—regulatory requirements. In some cases, you may need to use multiple approaches. Most programs initially measure savings based on deemed savings estimates or energy modeling and then verify these savings later using analysis of energy bills, most often based on a full year of billing after upgrades are completed. These three methods—deemed savings, energy modeling, and billing analysis—are described below.
Deemed savings are estimated energy savings from specific measures (e.g., insulation, duct sealing, etc.) based on historical evaluations of actual savings from measure installation. (Deemed savings are generally only calculated for measures with well–known and consistent performance characteristics). Deemed savings estimates can be derived from a number of sources, including past program evaluations, engineering estimates, and extrapolated results from other programs. The advantage of deemed savings is that they are straightforward to use and don’t require actual energy use monitoring. The disadvantages are that extrapolated savings may not be accurate for different climate zones or buildings, methodologies used to calculate deemed savings can differ among organizations calculating them, and they rarely account for the combined effects of multiple measures. The U.S. Department of Energy’s Uniform Methods Project is developing a framework and a set of protocols for consistent methodologies to determine the energy savings from specific energy efficiency measures and programs.
Utilities that make extensive use of deemed savings often document them in technical reference manuals. For example, utility energy efficiency program administrators in Massachusetts have developed a manual that describes how deemed savings will be calculated for energy efficiency programs in the state. It includes algorithms, default assumptions, and other information for calculating deemed savings from a variety of energy efficiency measures.
Energy modeling is often used for measures with savings that cannot be estimated with a single deemed value. These savings estimates may depend on a number of project-specific factors. For example, ceiling insulation savings will vary based on the R-value of insulation before and after an upgrade, the treated area, and the number of heating degree days. Some programs use home assessment software tools that have such algorithms embedded in them. Energy modeling software allows programs to estimate savings associated with single or multiple recommended measures.
Billing analysis provides estimates of savings based on statistical analysis of large datasets of participant and control groups. Typically a year or more of program participation data are needed before such analyses can be done. Results from billing analyses are often used to correct prior savings estimates derived from one of the two approaches above and/or to inform prospective deemed savings estimates. The Uniform Methods Project recommends billing analysis as the preferred approach for whole building residential energy efficiency programs.
EnergySmart Tracks Program MetricsBoulder County’s EnergySmart program established a customer relationship management system to track and display customer information, including deemed savings from upgrades. Among other metrics, it allowed the program to actively monitor overall program accomplishments and key activities:
- Total energy savings from all projects
- Total greenhouse gas reductions from projects
- Number of rebates awarded
- Cumulative number of upgrades
- Cumulative number of homes enrolled in the program
- The types of services provided to homeowners (e.g., energy advisor, assessment, upgrades)
- Customer motivations for participating in the program.
For more on EnergySmart’s approach, see the program’s presentation on using data to monitor its projects and impacts.
You will need to identify metrics for goals and objectives related to specific program components, such as marketing and outreach, financing, and contractor engagement & workforce development to track the progress made by those parts of your program.
You may need to establish metrics to demonstrate whether your program is meeting regulatory cost-effectiveness tests. Understanding these tests is important if your program is run by a utility, you partner closely with utilities in delivering a program, or you are otherwise impacted by state public utility commission regulation. If your program is not deemed cost-effective, you may not get funding unless you change your program design to deliver more energy savings at lower cost.
Key metrics used to assess cost-effectiveness include:
- Benefit/cost ratio
- Net benefits
- Total benefits
- Total cost.
These metrics can be measured using one or more of several cost-effectiveness tests:
- Participant cost test (PCT)
- Utility/program administrator cost test (PACT)
- Ratepayer impact measure test (RIM)
- Total resource cost test (TRC)
- Societal cost test (SCT).
Depending on the regulatory jurisdiction, application of cost-effectiveness metrics, particularly benefit/cost ratios, may need to be applied at different levels of program engagement, such as:
The cost-effectiveness targets may also vary depending on the level of program engagement. For example, while there may be a target benefit/cost ratio of 1.0 or greater at the portfolio and program level, not all jurisdictions require that individual measures be cost-effective—i.e., the benefit/cost ratio may be less than 1.0.For more information on cost-effectiveness tests, see the Evaluation & Data Collection handbook, Develop Evaluation Plans. The Existing Homes Program Guide by the Consortium for Energy Efficiency is another helpful resource.
Establish direct and indirect customer experience metrics
Like program metrics, customer experience metrics are an important way to understand if your overall program design is effectively delivering services to your customers. No residential energy efficiency program will achieve its energy savings and other goals if customers are not satisfied with the program (e.g., ease of participating in the program, responsiveness of program staff and contractors, value of program services). Customer experience metrics are also a valuable diagnostic tool for understanding where problems are developing and identifying ways to improve program design and delivery.
Direct Customer Satisfaction Metrics
Often based on phone or email surveys at the end of the upgrade process, direct customer satisfaction metrics capture overall satisfaction with a program (e.g., the number of customers rating a program as “high” or “very high” on a scale from very low to very high) as well as satisfaction with particular components (e.g., assessment or upgrade quality and value, contractor timeliness, program communications).
Templates and Examples of Customer Surveys
Customer satisfaction metrics can be developed from responses customer surveys—either using the response from a single question or “rolling up” responses to multiple questions into a single satisfaction score. The following sample customer surveys and survey questions were developed by the Better Buildings Neighborhood Program:
- Sample email survey template for successful program participants
- Research questions and survey questions for successful, drop-out and screened-out program participants.
- Sample phone survey template for energy efficiency program drop-outs
- Sample phone survey for applicants who have been screened out from participating in a residential energy efficiency program.
The following homeowner survey examples were developed by state and local residential energy efficiency programs:
- EnergySmart: Survey about a homeowner's experience with the program
- Minnesota Community Energy Services Survey for homeowners about the program
- Wisconsin Energy Conservation Corporation:
Indirect Customer Satisfaction Metrics
Indirect metrics help programs understand customer satisfaction indirectly, often with metrics that suggest sustained or increasing demand for program services. Examples include:
- The number of customers who refer the program to others
- The number of new customers who heard of the program from neighbors
- The number of customers who return for additional services.
Data for these types of metrics can be collected through surveys at the beginning of the upgrade process. Programs can also use website analytics to look for patterns that suggest increased demand for the program based on customers seeking certain types of information on a program website (e.g., information about upgrade contractors).
Document measurement strategies for collecting data
Once you have identified metrics, document how they will be measured. The table below, drawn from the Better Buildings Neighborhood Program’s “Creating an Evaluation Plan” worksheet illustrates a simple way to document measurement strategies for program metrics linked to program goals. Although the presentation in the table is simple, some of the methods for measuring metrics can be quite complex. The Evaluation & Data Collection Develop Resources handbook provides a detailed description of types of evaluation metric data and how to collect and assess them.
Source: U.S. Department of Energy, 2014.
Two other parts of the strategy that you need to identify include:
- Who will provide or collect the data (e.g., program staff, contractors, financial partners)?
- What quality assurance checks on the data will be required and how they will be conducted? For example, data can be automatically or manually checked against a range of expected values to check for outlier data points.
Once measurement strategies have been identified, program staff, contractors, and others responsible for collecting data should be trained on what data will be collected, at what frequency, and how to report it.
Establish responsibilities for internal review of evaluation results
As part of evaluation planning, programs should establish responsibilities and schedules for review of evaluation results. Because evaluation is critically important, someone needs to take responsibility for it—otherwise, it is much less likely to receive the attention it needs. Key responsibilities are:
- Periodically check program metrics against overall goals and objectives
- Understand the level and trends in program activity
- Monitor customer satisfaction as an indicator of the overall health of the program
- Identify successes or challenges that can be used to assess and improve the program.
An example of program positions and responsibilities is shown in the table below.
Source: U.S. Department of Energy, 2014.
The process of internal review and decision-making to respond to evaluation results is discussed further in the handbook on assessing and improving programs and in the Evaluation and Data Collection handbook on communicating program impacts.
Tips for Success
Set realistic expectations for launching and scaling up your program
Many program administrators have found that launching and scaling up a program often takes longer than planned for, especially when forming partnerships with contractors and lenders. New energy efficiency programs often need at least 2-3 years to launch and become fully operational. Across programs, the amount of time it takes to get to full operations depends on many factors, including the number of qualified contractors working in the area, the availability of funding, the level of stakeholder and partner support that is available, the program’s goals and strategies, and the presence of unique program features that may take time to develop, such as community workforce agreements or loan products. Many program administrators found it helpful to set realistic expectations internally—and with key partners and stakeholders—about how long it takes to get programs fully up and running. And, they suggest celebrating and communicating achievements along the way.
- emPowerSBC in Santa Barbara, California, found that launching its program and scaling up took more time than expected. The launch of the program was delayed more than a year as the program modified its financing strategy from one that relied on residential PACE to one focused on a loan-loss reserve. Following the launch, hiring delays kept the program from being fully staffed for around six months. Contractors working with the program reported that it took three to twelve months for a lead to turn into a signed contract for upgrade services because homeowners took their time deciding whether to invest in energy efficiency.
- The Virginia State Energy Program (SEP) found that it was difficult for its three programs around the state—the Local Energy Alliance Program (LEAP), the Richmond Region Energy Alliance, and Community Alliance for Energy Efficiency (Cafe2)—to meet their upgrade targets in three years because the home performance industry in the state was still developing when the programs were initiated. These in-state programs started with little to no infrastructure in place and had to address barriers such as lack of qualified contractors before they could even begin offering home energy upgrades. For example, the programs found that contractors were reluctant to modify their business models and agree to undertake the paperwork and data collection the programs required. Over time, the programs developed strategies to work more effectively with contractors, such as holding monthly contractor meetings (in the case of LEAP) and establishing written Memoranda of Understanding with contractors to clarify mutual expectations (in the case of Cafe2). Virginia SEP advised that programs’ goals and timelines should reflect the starting conditions and the work that needs to be accomplished in order to achieve program goals.
- Enhabit, formerly Clean Energy Works Oregon, began with modest goals for a pilot project in Portland and then ramped up its ambitions as it expanded statewide. The goals for the program’s pilot project were to upgrade 500 homes in Portland, build a qualified workforce, and test its approach to service delivery. After the pilot, the program expanded to all of Oregon and upgraded over 3,000 homes around the state in three years.
Measure and evaluate performance at key points in the process
Measuring performance at key points in the upgrade process (e.g., assessments, conversion rates, and financing applications) has helped programs understand where their processes are working smoothly and where they are not. This information has helped them continuously improve their program design and implementation. To monitor progress, successful programs have combined information from their project tracking systems with customer surveys, information from call centers, and feedback from contractors and lenders to understand the customer experience. Make data accessible for program staff to track progress, identify successful strategies, and detect points of failure.
- Enhabit, formerly Clean Energy Works Oregon, established an extensive process for getting customer feedback at key points in the program delivery process to evaluate customer satisfaction and better understand why some homeowners chose to undertake upgrades while others did not. The program identified seven points in the program delivery process to gather information through feedback surveys and phone interviews: application, assessment, bid, drop-out, financing, completion, and experience after 12 months. The program credited this kind of customer communication and feedback as one of the keys to its ongoing success.
Source: Clean Energy Works Research Planning, Will Villota, CEWO, 2012 (Presented during January 19, 2012 Better Buildings Residential Neighborhood Program peer exchange call).
- Boulder County’s EnergySmart program sent an online customer feedback survey to homeowners who had completed upgrades. Among other things, the customer surveys affirmed customer satisfaction and identified the opportunity for word-of-mouth marketing. Surveys found that the vast majority of the respondents would recommend the EnergySmart service to a friend or neighbor. The surveys also surfaced some weaknesses that the program resolved. For example, some respondents noted contractor’s lack of response and professionalism as an issue, which led the program to develop guidelines for professionalism and customer contact. Surveys also noted that the assessment report was long and confusing, leading the program to develop a new, customized report that was easier to follow and clearer about next steps.
- Connecticut’s Neighbor to Neighbor Energy Challenge used qualitative contractor and customer feedback combined with quantitative data to evaluate how well its outreach efforts led to home energy assessments. When informal contractor feedback alerted program managers that relatively few interested customers were following through to have assessments conducted on their homes, the program analyzed project data and found that only around a quarter of customers who expressed interest in an assessment had completed one. To diagnose the problem, the program analyzed data to see how customers were acquired, how long it took to send leads to contractors, and how long it took contractors to follow up with customers to arrange for an assessment. Through qualitative analysis, the program found, among other things, that customers didn’t understand what they were signing up for and may have been unwilling to say “no” to young and enthusiastic outreach staff. The program also found that its staff wasn’t following up quickly enough with people that wanted more information. In response, the program improved its process for distributing leads to contractors (e.g., linking contractors to homeowners in 1-2 days), created a “receipt” for interested customers outlining next steps, and set up a system to call non-responsive leads after two weeks. With these and other steps, the program increased its close rate 35% in one month after changes were implemented.
Last Updated: 01/20/2016