U.S. Department of Energy Energy Efficiency & Renewable Energy U.S. Department of Energy Energy Efficiency & Renewable Energy

Description

Key Resources

As the manager of an energy efficiency program, you need to be asking questions about the performance and outcomes of your program:

  • How well is the program accomplishing its objectives?
  • How effective are the marketing campaigns?
  • How satisfied are participants with the available incentives, including any financing offered?
  • Just how much of a difference are we making in the local home improvement market?
  • Is the program cost-effective?
  • How can we improve program performance?

Questions like these—about performance and outcomes—ultimately involve evaluation of the effects of a program.

There are three common types of energy efficiency program evaluations: impact, process, and market evaluations. These can include a wide range of assessment studies to determine the effects of a program.

Common Types of Energy Efficiency Program Evaluations

The three evaluation types are not mutually exclusive. While their focus may differ, they each have important interconnections.

Impact evaluation: An evaluation of the program-specific changes (e.g., changes in energy and/or demand use), directly or indirectly induced, associated with an energy efficiency program.

Impact evaluations can help you:

  • Determine the extent to which your program has achieved the projected energy savings.
  • Understand the non-energy benefits (or co-benefits) such as avoided emissions and job creation that directly or indirectly result from a program.
  • Quantify program outcomes for use in communications and marketing and outreach.

Process evaluation: A systematic assessment of an energy efficiency program. Their purpose is to document program operations and identify and recommend improvements to increase the program’s efficiency or effectiveness for acquiring energy resources while maintaining high levels of participant satisfaction.

Process evaluations can help you:

  • Identify the extent to which existing processes and practices are, or are not, contributing to your program’s performance.
  • Identify ways to improve program performance.

Market evaluation: An evaluation of the change in the structure or functioning of a market, or the behavior of participants in a market, that results from one or more program efforts. Typically, the resultant market or behavior change leads to an increase in the adoption of energy efficient products, services, or practices.

Market evaluations can help you:

  • Gain insights into the relevant market for your program.
  • Identify how the market or market participants have changed as a result of your program.
  • Understand what approaches had a positive effect on the market.

Source: Energy Efficiency Program Impact Evaluation Guide, State and Local Energy Efficiency Action Network, 2012.

Although active program management will involve data collection and analysis for continuous improvement, evaluations of impact, process, and market effects are typically conducted independently from day-to-day program management. These evaluations may be conducted by a third-party evaluator or an internal team.

Results from third-party evaluations not only provide actionable information for program improvement, they also credibly demonstrate results to funders, stakeholders, and the public. For example, a program that can show its cost-effectiveness—in terms of progress toward goals compared to program expenditures—is more likely to be viewed as successful by appropriators and stakeholders.

Internal process optimization is performed in-house and more often than formal, third-party process evaluations, so that program managers can make timely changes to improve the program. Routine internal evaluation is a critical program management tool that will help improve program performance, and thus improve the outcomes of third-party evaluations. See the Program Design & Customer Experience handbook for information on how to build internal evaluation into program management.

This handbook introduces Evaluation & Data Collection and leads to a series of handbooks exploring this area in detail. Taken together, these handbooks provide guidance for program administrators on:

  1. Plan evaluations, focusing on external evaluations conducted by third-party evaluators. Topics include setting the scope, schedule, and budgets; designing appropriate data collection systems and processes; developing appropriate cost-effectiveness criteria; and engaging evaluators and other partners. This handbook also discusses how to integrate evaluation with all program aspects, especially the daily activities that affect data collection quality and relevance. Internal evaluations are covered in the Program Design & Customer Experience handbooks.
  2. Develop the resources needed to support external evaluation activities, including implementation of tools and systems that will support data collection and quality assurance necessary for effective evaluation, and structuring of data release agreements.
  3. Conduct and manage evaluation activities, including working with external evaluation partners and other stakeholders, as well as with all internal organization actors with a role in affecting evaluation activities and quality assurance.
  4. Communicate evaluation results to contractors, funders, regulators, the public, and other audiences. This handbook discusses how to use third-party evaluations, particularly impact evaluations, to help positively influence these stakeholders and provide necessary validation of the program.

Better Buildings Neighborhood Program Evaluation

To understand how the Better Buildings Neighborhood Program affected the residential energy efficiency market and industry, the U.S. Department of Energy worked with third-parties to conduct a comprehensive impact, process, and market effects evaluation. The full suite of evaluation reports, completed in June 2015, include: 

  1. Evaluation of the Better Buildings Neighborhood Program (Final Synthesis Report Volume 1)
  2. Savings and Economic Impacts of the Better Buildings Neighborhood Program (Final Evaluation Volume 2)
  3. Drivers of Success in the Better Buildings Neighborhood Program—Statistical Process Evaluation (Final Evaluation Volume 3)
  4. Process Evaluation of the Better Buildings Neighborhood Program (Final Evaluation Volume 4)
  5. Market Effects of the Better Buildings Neighborhood Program (Final Evaluation Volume 5)
  6. Spotlight on Key Program Strategies from the Better Buildings Neighborhood Program (Final Evaluation  Volume 6)

For more information about the evaluation, see the Better Buildings Neighborhood Program Accomplishments webpage. 

 

These handbooks provide more detail on designing and implementing a cohesive and balanced residential energy efficiency program.
  • Identify your organization's preferred market position by assessing existing market actors, gaps, competitors, and potential partners.  Develop a business model that will allow you to deliver energy efficiency services.

  • Design a residential energy efficiency program that integrates marketing and outreach, contractor coordination, incentives, financing, and program evaluation to provide customers with the products and services they want through a customer-centric process.

  • Spur consumer demand for your program's services by understanding your target audience and motivating them to act using effective messaging, marketing and outreach tactics, and attractive program offers.

  • Ensure that your program’s customers will have access to affordable financing, so they can pay for the services you offer.

  • Support and partner with the workforce who will deliver your program’s energy efficiency services by understanding their capacity, recruiting contractor partners, enabling technical training and business development support, fostering clear communication, and refining program processes over time, in partnership with your workforce.

Step-by-Step

The following are important stages for successful program administrators to follow when implementing Evaluation & Data Collection activities; however, no two programs are the same, and program administrators need to take into account the unique aspects of their market to create the most effective approach possible. Select each stage to access its handbook.

  1. Develop Evaluation Plans
    Identify the right questions to ask, appropriate metrics to collect, and the processes needed to initiate third-party impact and process evaluations.
  2. Develop Resources
    Identify and implement systems and tools that will support data collection and data quality necessary for effective evaluation.
  3. Conduct Evaluation
    Manage third-party impact and process evaluation activities by coordinating with evaluators, transferring data, and overseeing evaluation deliverables.
  4. Communicate Impacts
    Communicate pertinent results of evaluations to program staff, partners, and stakeholders.

Tips for Success

In recent years, hundreds of communities have been working to promote home energy upgrades through programs such as the Better Buildings Neighborhood Program, Home Performance with ENERGY STAR, utility-sponsored programs, and others. The following tips present the top lessons these programs want to share related to this handbook. This list is not exhaustive.

Develop data collection and evaluation plans in conjunction with program design

Many Better Buildings Neighborhood Program partners found that it was important to communicate during the program design phase with organizations and individuals that will collect or supply data for the evaluation. In this way, the involved individuals and organizations understand why the data is being collected and are invested in the process. Also, involving evaluators in the design of the program ensures that the right data is collected from the start.

  • The Southeast Energy Efficiency Alliance (SEEA) programs experienced challenges collecting the information needed for third-party evaluation, measurement, and verification activities. SEEA contracted a third party, Cadmus Group, to evaluate its programs’ activities, and the evaluator found that the data they used as inputs - programmatic data from the Quarterly Program reports to DOE and utility bill data to perform their analysis - did not always correlate entirely with the data that programs were collecting from homeowners and contractors. SEEA and Cadmus worked with each of the programs to fill in the gaps. In some cases the programs were able to provide this information. For others, they were unable to because it only existed in paper format or the program did not have the time or resources to provide the information.  To address this issue, Cadmus developed a flexible approach to determine savings for each program, based on database reviews, billing analysis, technical desk reviews, whole-house energy modeling, and engineering analysis.
    • Cadmus conducted technical desk reviews when sufficient billing data was not available, but when SEEA provided data files from simulation models or other project documentation for a sufficient sample of participant homes.
    • They used whole-house energy modeling to assess the reported savings when sufficient billing or simulation data was not available, but program tracking data was complete enough to populate required model inputs.
    • Engineering analysis was used to estimate savings for measures not incorporated into the models or to evaluate the applicability of data from other program locations when little or no data other than the program tracking data was available.
    • When data was not available, Cadmus compared weather and participant profiles to other program locations for similarities.
  • For EnergyWorks in Philadelphia, problems with data tracking became apparent during the data analysis conducted for the program evaluation. EnergyWorks determined that these problems could have been avoided by clearly articulating and communicating data collection goals to program staff and defining key program performance indicators early in the program. The program established an overarching goal of 2,000 upgrades but found that it would have also been good to have set smaller specific goals tied to performance in order to better understand trends as they were occurring over the life of the program and how these trends were impacting the overall goal of 2,000 upgrades.
Close

Establish data sharing relationships as early as possible

Though potentially challenging, establishing relationships for sharing energy consumption data is critical for evaluating program impact on energy and cost savings. Many Better Buildings Neighborhood Program partners found success by approaching utilities during the program planning phase, or at least several months in advance of when they planned to start collecting data, to outline shared goals, assets, tools, needs and constraints. Clear and concise data requests helped speed up utilities’ response times for providing the data and alleviated utility concerns and questions regarding data needs.

  • Energize Phoenix formed a partnership with the local electric utility, Arizona Public Service, while designing the program and coordinated with them throughout program development. Energize Phoenix found that understanding Arizona Public Service’s concerns and challenges related to data sharing was a key ingredient in forging a successful partnership, as was instituting a formal agreement to clarify roles and responsibilities.
  • Southeast Energy Efficiency Alliance (SEEA) found that not all of their programs were successful in obtaining utility bill data. Common obstacles included that the utility did not have the technology infrastructure to easily export the information, would only release data for a fee (based on how many records were pulled), or simply did not have the time or resources to provide the information even if the program had a signed client release form from the homeowner. Among SEEA's programs, those that were most successful in obtaining utility billing information–including NOLA WISE in New Orleans, Louisiana; Local Energy Alliance Program (LEAP) in Charlottesville, Virginia; Atlanta SHINE in Atlanta, Georgia; and DecaturWISE in Decatur, Georgia - consulted with the utilities to determine what information the program needed to include in the client release form. Additionally, some programs developed a written memorandum of understanding with the utility specifying data collection and transfer roles and responsibilities. SEEA programs also found it best to make data requests to utilities on a quarterly basis to minimize the burden on the utility as many utilities do not have staff dedicated to data exporting. Some programs received data more frequently, but in these situations the utility had the means to easily pull and export data.
  • When local utilities Philadelphia Gas Works (PGW) and Philadelphia Electric Company (PECO) shared customers’ energy usage data with EnergyWorks, all parties made sure that the proper data sharing requirements were observed and all parties signed the necessary forms. Philadelphia EnergyWorks built its customer data release approval language into the program’s loan application form to minimize the number of additional forms that a customer or contractor would need to handle.
  • EnergySmart in Eagle County, Colorado, successfully developed partnerships with utilities during and after the Better Buildings Neighborhood Program grant period, but in hindsight found it would have been more beneficial to engage utilities prior to submitting the original DOE grant application. By not fully engaging utilities up front, EnergySmart created the following  environment where the utilities are only partially included in the program and retained similar or redundant in-house services. As EnergySmart continued forward, they were able to gain the trust of the utility by offering help, data, and information. EnergySmart also shared their results with the utility’s management and board of directors. Through this gained trust, utilities were more willing to share data.
Close

Invest in information and communications technology

Paper-based or spreadsheet-based information collection processes can be low cost to develop and easy to roll-out, but more often than not, they become cumbersome to aggregate and store the data from many sources. Many Better Buildings Neighborhood Program partners found that investing in information and communications technologies (ICT)  eased program implementation and was well worth the effort because they were able to regularly monitoring progress and automate what would otherwise be time-intensive, manual processes. For more information on the future of ICT, see ACEEE’s How Information and Communications Technologies Will Change the Evaluation, Measurement, and Verification of Energy Efficiency Programs.

  • Garfield Clean Energy in Garfield County, Colorado, at first used a series of Excel spreadsheets and hardcopy file folders to track participants, their energy upgrade measures, and resulting energy savings. As the number of participants reached into the hundreds, the program realized that spreadsheets did not offer the level of sophisticated searching and reporting that was needed to analyze the results of their work. They explored several online customer relationship management systems and contracted with a third-party developer to customize their selected system so that it could track building and energy data, energy upgrades, contractors, dollars spent, rebates awarded, and deemed energy savings. The customization and data entry work, which took several months to complete, enabled Garfield Clean Energy to create detailed reports based on a wide variety of reporting parameters, and to better analyze the effectiveness of program activities.
  • When Enhabit, formerly Clean Energy Works Oregon, scaled up their pilot program to the entire city of Portland, it was clear to them that an IT solution was needed to meet the demands of funding agencies, media requests, and good project management from the customer perspective. Enhabit worked with software company EnergySavvy to develop a unified service delivery platform to manage the home energy upgrade process from application to completion. The software platform provides a web-based interface between homeowners, contractors, and lenders, enabling each party to document progress through the Enhabit program. The platform also streamlined data collection and analysis.
  • In Boulder, Colorado, EnergySmart used spreadsheets to manage data for its predecessor energy efficiency programs. As the program expanded under the Better Buildings Neighborhood Program, it became clear that EnergySmart needed to pursue a more user-friendly, real-time, cloud-based IT system for tracking customers through the implementation process. They selected a system to allow for tracking of many metrics in a much more consistent, accurate, and organized fashion than the previously used spreadsheet. The system can be accessed in the field by EnergySmart Energy Advisors using iPads or tablets to enter basic customer information, building baseline information, assessment findings for upgrade opportunities, completed upgrades with associated energy and cost savings, rebates and financing received, and the supporting documentation. The collected data is compiled for reporting to various stakeholders, including the U.S. Department of Energy, county commissioners, and city staff and leaders. The ability of Energy Advisors to access the system in the field allows for much greater efficiency and accuracy than the static logging of data upon returning to the office.
Close

Provide adequate time for data system development and testing

Many Better Buildings Neighborhood Program partners found that setting up their information technology (IT) systems early in the program design stage ensured that data terms and data entry procedures were consistently applied by all system users. Reaching agreement with stakeholders (e.g., contractors, lenders, marketing partners, evaluators, program staff) on what data the data system would collect, known as system requirements, and how the collected data would be used to evaluate the program helped programs ensure that the data collected was complete. Programs have also found that they receive data of the quality needed for graphs and cost-effectiveness calculations when stakeholders agree up front that the data will be used for these purposes and not just to track energy savings and expenditures.

  • Be SMART Maryland found that transitioning from spreadsheet-based data collection system to a customized energy IT system was crucial to administering a multifaceted energy efficiency program with rigorous data collection requirements. Investing in their system while they were still designing their program allowed Be SMART to smoothly integrate the system into the program’s operations and to ensure quality data collection and integrity over time. Be SMART also found that while spreadsheets were useful tools in collecting data, their use in analyzing data and generating reports was limited, since the program had to go through a time-consuming consolidation process to combine data from different sources and spreadsheets.
  • In Boulder County, Colorado, EnergySmart found that it took between four to six months for a database developer and coding consultant to fully develop and test the data system because of its high level of complexity and the customization required. The program also found that having actual users test the system with real inputs and real reporting requirements helped ensure better data quality and user-friendliness. In addition, EnergySmart found that before beginning database development, it was important to reach agreement among stakeholders on what reporting will be expected, and design the database to facilitate building and exporting the reports. For EnergySmart, it was important to set expectations with report recipients about the IT system’s reporting capabilities early on in the process, so recipients did not expect reports that the system was unable to produce.
Close

Use compatible formats for data sharing and reporting, and work with partners to implement standard data exchange protocols

Many Better Buildings Neighborhood Program partners found that it was critically important to use compatible formats for data sharing and reporting with partners. Aligning data formats and collection plans with national data formats (e.g., Home Performance XML schema (HPXML), Standard Energy Efficiency Data platform (SEED), Building Energy Data Exchange (BEDES)) ensured compatibility for aggregation and reporting.

  • For Arizona Public Service’s (APS) Home Performance with ENERGY STAR® Program, a lack of transparency and access to data meant it took hours each month to compile progress reports. Coordination with trade allies was difficult for similar reasons–both the utility and its contractors lacked visibility into project status and task assignment, as well as the ability to identify program bottlenecks, which impacted APS customer service. Program delivery metrics, from administrative overhead to customer and trade ally satisfaction, were lower than expected. APS then began the search for a more dynamic software platform to engage customers, track and manage projects, empower trade allies, and analyze and report results. The program needed HPXML, an open standard that enables different software tools to easily share home performance data. The new HPXML-compliant platform, EnergySavvy’s Optix Manage, resulted in higher cost effectiveness and greater satisfaction for the program, including 50% less administrative time to review and approve projects, a 66% reduction in data processing time for APS reporting, 31% less contractor administrative time to submit projects, and a three-fold increase in trade ally satisfaction.  HPXML also had the added benefit that contractors can choose their own modeling software.
  • The New York State Energy Research & Development Authority (NYSERDA) heard from home performance contractors and other stakeholders that a more streamlined data collection process was needed to reduce the paperwork burden and time spent on a per project basis. In response, the program launched the NY Home Performance Portal in July 2013. This web-based interface made it easier for customers to choose and apply for the home performance program and made the application process for a home energy assessment clear, fast, and simple. In 2015, NYSERDA further refined their data collection process and began processing of all projects in a web-enabled interface designed to facilitate program coordination. This new platform allowed NYSERDA to automate project approvals for 85-90% of projects. In addition, the platform supported HPXML which facilitates data sharing among multiple New York programs, thereby reducing the administrative burden for contractors participating in multiple programs. It allowed NYSERDA to automate the work scope approval process through validation of standardized data. An additional benefit of HPXML for NYSERDA was creating an open modeling software market.
  • Massachusetts Department of Energy Resources (MassDOER) provides statewide oversight to energy efficient programs administered by utilities under the Mass Save brand. Originally, contractors from Conservation Services Group, Inc. and Honeywell International Inc. used audit software customized for the program in their home energy assessments.  When MassSave piloted the Home MPG program, contractors were also required to generate an Energy Performance Scorecard for each home. The existing audit software, however, did not have this capability. To address this problem, software developers added the Energy Performance Scorecard capability, so the contractors could use the same software to record the audit results and generate the energy performance scorecard. Despite implementation delays, this solution allows the use of the Energy Performance Scorecards to potentially expand to statewide.
Close

Examples

The following resources are examples from individual residential energy efficiency programs, which include case studies, program presentations and reports, and program materials. The U.S. Department of Energy does not endorse these materials.

Case Studies

None available at this time.

Program Presentations & Reports

  1. Author: Marlowe Kulley, Portland Bureau of Planning & Sustainability
    Publication Date: 2011

    This presentation is a tour of the project evaluation and data collection system that Clean Energy Works Portland uses to survey its participating residents.

  2. Author: Melissa Glickman, Boulder County, Colorado (now EnergySmart)
    Publication Date: 2012

    EnergySmart Colorado uses surveys and a customer database to get feedback from homeowners that helps fine-tune program services and operations.

  3. Author: Betsy Kleinfelder, The Sustainability Institute
    Publication Date: 2012

    As part of its "intentional learning" process, Charleston WISE collects information from homeowners that helps the program systematically test assumptions and implement continuous improvement.

  4. Author: Dimitrios Laloudakis, Energize Phoenix
    Publication Date: 2011

    This presentation outlines the techniques for collecting and evaluating energy efficiency program evaluation data, including data related to marketing efforts.

  5. Author: U.S. Department of Energy
    Publication Date: 2012

    A list of tips from Connecticut's Neighbor to Neighbor Energy Challenge for working with utilities to access energy usage data.

  6. Author: Southeast Energy Efficiency Alliance
    Publication Date: 2013

    This report provides an independent analysis of the job creation impact of DOE's investment in energy efficiency programs, from 2010 to 2013. The analysis calculates the job creation results that would have occurred in the Southeast, based on the prevailing economic conditions from 2010 to 2013, had DOE invested in sectors other than energy efficiency.

Program Materials

  1. Author: RePower Bainbridge
    Publication Date: 2012

    Homeowner data collection survey created by RePower.

Toolbox

The following resources are available to help design, implement, and evaluate possible activities related to this handbook. These resources include templates and forms, as well as tools and calculators. The U.S. Department of Energy does not endorse these materials.

Templates & Forms

  1. Author: U.S. Department of Energy
    Publication Date: 2015

    The Evaluation & Data Collection Implementation Plan Template will help you develop a strategy for planning, operating, and evaluating your data collection and evaluation activities.

Tools & Calculators

  1. Author: National Home Performance Council
    Publication Date: 2012

    A pro forma is a tool of forecasting the impact that adjustments to a business model can have on future financials, using a set of assumptions and inputs. In the residential energy efficiency industry, programs can use pro forma tools to forecast the impact that marketing campaigns, incentive re-structuring, or other program changes will have on the program budget and results.  Example assumptions include the number of homeowner registrations that a set of marketing activities generate in a year, average assessment to upgrade conversion rate, and average incentive per project. By applying assumptions such as these, a pro forma tool can also help your program determine how effective various strategies are at achieving program goals and objectives. Program administrators can help contractors by supporting them with their own business pro forma. To help you get started, here are a few useful resources: the National Home Performance Council developed a presentation on their Integrated Pro Forma Project; for an example program pro forma, see the presentation by Virginia’s Local Energy Alliance Program (LEAP-VA); the National Home Performance Council also developed the Contractor Pro Forma Tool.

  2. Author: U.S. Department of Energy

    The Building Energy Data Exchange Specification (BEDES, pronounced "beads" or /bi:ds/) is designed to support analysis of the measured energy performance of commercial, multifamily, and residential buildings, by providing a common data format, definitions, and an exchange protocol for building characteristics, efficiency measures, and energy use.

  3. Author: Building Performance Institute
    Publication Date: 2012

    Home Performance Extensible Markup Language (HPXML) is a data transfer protocol. It was developed to simplify electronic data transfer between any party involved in a home performance program, including contractors, program administrators, utilities, and federal agencies.

  4. Author: U.S. Department of Energy
    Publication Date: 2014

    The Standard Energy Efficiency Data (SEED)™ Platform is a software application that helps organizations easily manage data on the energy performance of large groups of buildings. Users can combine data from multiple sources, clean and validate it, and share the information with others. The software application provides an easy, flexible, and cost-effective method to improve the quality and availability of data to help demonstrate the economic and environmental benefits of energy efficiency, to implement programs, and to target investment activity.

  5. Author: U.S. Environmental Protection Agency
    Publication Date: 2012

    A comprehensive source of data on the environmental characteristics of almost all electric power generated in the United States.

  6. Author: U.S. Environmental Protection Agency
    Publication Date: 2012

    A tool that provides information on the air emissions attributable to the electricity used in a home or business during one year, along with a description of what these numbers mean in everyday terms and information on how to be more energy efficient or buy green power.

  7. Author: U.S. Department of Energy

    The Buildings Performance Database (BPD) is the largest national dataset of real building performance data, and enables users to perform statistical analysis on an anonymous dataset of hundreds of thousands of commercial and residential buildings from across the country. One of the most powerful applications of the tool is custom peer group analysis, in which users can examine specific building types and geographic areas, compare performance trends among similar buildings, identify and prioritize cost-saving energy efficiency improvements, and assess the range of likely savings from these improvements.

Topical Resources

Topical Presentations

  1. Author: Jane Peters, Research Into Action, Inc.
    Publication Date: 2010

    This presentation covers the importance of collecting and evaluating program data, including data related to marketing efforts.

  2. Author: Cecily McChalicher, Northeast Energy Efficiency Partnerships, Inc.
    Publication Date: 2013

    This presentation includes a brief Introduction to Northeast Energy Efficiency Partnerships (NEEP) and the Regional Evaluation, Measurement and Verification Forum (EM&V Forum), Regional Energy Efficiency Database (REED) development and content, and how to access the REED reports and underlying data.

  3. Author: Patrick Roche, Conservation Services Group
    Publication Date: 2012

    Presentation describing how Conservation Services Group uses data to monitor market transformation and for internal QA/QC purposes.

  4. Author: Lawrence Berkeley National Laboratory
    Publication Date: 2011

    A process guide for taking open-ended survey questions and generating actionable items for program improvement.

Publications

  1. Author: U.S. Department of Energy
    Publication Date: 2015

    Volume 1 of the Better Buildings Neighborhood Program Evaluation Report provides findings from a comprehensive impact, process, and market effects evaluation of the program period, spanning from September 2010 through August 2013.

  2. Author: U.S. Department of Energy
    Publication Date: 2015

    Volume 2 of the Better Buildings Neighborhood Program Evaluation Report comprises a measurement and verification process, as well as billing regression analysis on projects with sufficient utility bill data, to determine gross verified savings.

  3. Author: U.S. Department of Energy
    Publication Date: 2015

    Volume 3 of the Better Buildings Neighborhood Program Evaluation Report statistically identifies factors associated with successful residential energy upgrade programs using a survey sampling, cluster analysis, and multivariate regression approach.

  4. Author: U.S. Department of Energy
    Publication Date: 2015

    Volume 4 of the Better Buildings Neighborhood Program Evaluation Report assesses the degree to which the Better Buildings Neighborhood Program met its process goals and objectives to identify the most effective program design and implementation approaches.

  5. Author: U.S. Department of Energy
    Publication Date: 2015

    Volume 5 of the Better Buildings Neighborhood Program Evaluation Report provides findings from a comprehensive impact, process, and market effects evaluation of the program period, spanning from September 2010 through August 2013.

  6. Author: U.S. Department of Energy
    Publication Date: 2015

    Volume 6 of the Better Buildings Neighborhood Program Evaluation Report provides findings from a comprehensive impact, process, and market effects evaluation of the program period, spanning from September 2010 through August 2013. This volume includes case studies that describe successful strategies that programs used during the evaluation period.

  7. Author: State and Local Energy Efficiency Action Network
    Publication Date: 2012

    This guide provides an introduction to the key issues, practices, and steps for calculating energy savings, avoided emissions, and other non-energy impacts associated with energy efficiency programs.

  8. Author: National Home Performance Council
    Publication Date: 2012

    This report provides a comprehensive review of a wide range of problems and inconsistencies in current cost-effectiveness test practices, and recommends a range of best practices to address them.

  9. Author: National Home Performance Council
    Publication Date: 2012

    This paper describes the problems and issues that arise for energy efficiency programs as a result of common cost-effectiveness test implementation practice. It also provides recommendations for how to address these challenges.

  10. Author: U.S. Energy Information Administration
    Publication Date: 2013

    For this inventory, EIA reviewed and catalogued 329 data sources containing state energy efficiency program evaluation results into an inventory. The focus of this inventory is to support the National Energy Modeling System (NEMS) and to research cost information in state-mandated energy efficiency program evaluations.

  11. Author: Institute for Electric Efficiency
    Publication Date: 2013

    This report summarizes ongoing and recent policy developments that support utility investments in energy efficiency, including program cost recovery, fixed cost recovery, and performance incentives for electric utilities on a state-by-state basis.

  12. Author: State and Local Energy Efficiency Action Network
    Publication Date: 2012

    This report provides guidance and recommendations to help residential energy efficiency programs to more accurately estimate energy savings. It identifies steps program managers can take to ensure precise savings estimates, apply impact estimates over time, and account for and avoid potential double counting of savings.

  13. Author: American Council for an Energy-Efficient Economy
    Publication Date: 2015

    Information and communications technologies (ICT) can automate and transform the evaluation, measurement, and verification (EM&V) of energy efficiency programs. ICT enables the remote monitoring and sophisticated analysis of energy usage, increasing the speed and scale of many EM&V activities. This report reviews traditional EM&V practices, explores new enabling technologies including the Internet of Things and remote building analysis, and describes the application of ICT to each stage of the EM&V process. The report then projects ways forward through a number of challenges (e.g., data overload) and concludes that ICT-enabled EM&V could eventually change the design of efficiency programs and the responsibilities of program administrators, implementers, and evaluators.

Webcasts

  1. Author: U.S. Department of Agriculture; U.S. Department of Energy
    Publication Date: 2014
    Presentation, Media, Transcript

    This webinar is the first (in a series of six) hosted by USDA Rural Utility Service (RUS) and focusing on the Energy Efficiency and Conservation Loan Program (EECLP). This webinar provides an overview of the Energy Efficiency and Conservation Loan Program. It covers the requirements and benefits of the program and also discusses steps you can take to evaluate the cost effectiveness of energy program options.

  2. Author: U.S. Department of Agriculture; U.S. Department of Energy
    Publication Date: 2014
    Presentation, Media, Transcript

    This webinar is the second (in a series of six) hosted by USDA Rural Utility Service (RUS) and focusing on the Energy Efficiency and Conservation Loan Program (EECLP). This webinar covers the key concepts of Evaluation, Monitoring & Verification (EM&V), gives an overview of the full process, from estimating savings before programs are implemented to measuring and verifying the savings at the end. The webinar also covers EM&V framework, evaluation plans, technical reference manuals and measurement and verification studies.

  3. Author: Julie Michals, Northeast Energy Efficiency Partnerships, Inc.; Phil Sieper, The Cadmus Group, Inc.; Mark Stetz, Stetz Consulting
    Publication Date: 2010
    Presentation, Media
    (43 MB)
    , Transcript

    This webinar offers an introduction to EM&V basics, including data collection, tracking tools, M&V approaches, and reporting energy savings.

Quick Links

Browse key topics that many residential energy efficiency programs need to address. Select a Quick Link below to see curated resources, including case studies, presentations, tools, calculators, templates, and more. See a full list of all Quick Links (link to Quick Links landing page). If you have suggestions for additional Quick Links, please tell us.

Community events allow for face-to-face contact with target audiences and provide a venue to deliver messages to potential customers by trusted sources.

Community-based social marketing aims to change the behavior of a target audience within a group, neighborhood, or town based on principles of behavioral psychology and community dynamics.

Cost-effectiveness tests compare the benefits of a utility or non-utility program’s investment in energy efficiency with its associated costs. The five most common tests used by public utility commissions are: the participant cost test (PCT), the utility/program administrator cost test (PACT), the ratepayer impact measure test (RIM), the total resource cost test (TRC), and the societal cost test (SCT).

Customer engagement tactics aim to increase the likelihood that a customer will participate fully in a program by forming a relationship between the customer and the program or contractor. From pledges (which provide contractor leads and engage potential customers without requiring a full commitment from them) to testimonials (which encourage homeowners to share their upgrade experience with others), engaging customers in multiple ways can help increase program successes.

Establishing written data collection agreements will clarify data ownership, privacy, permission, types of data, and procedures for transferring data a program will obtain. Data collection agreements signed by homeowners are typically required for utilities to release billing data to a program or contractor; programs might also need to establish a formal agreement with lenders for any kind of data sharing. Program administrators are responsible for protecting consumer privacy and ensuring that all relevant parties use the appropriate forms, obtain data owner permissions, and avoid the release of confidential data.

Data exchange specifications help facilitate the transfer of data between software systems used by a program and its partners or stakeholders. A standard specification for transferring data reduces the need to develop a data transfer protocol each time two systems need to transfer information. Home performance XML (HPXML) is one emerging example for transferring data collected during an in-home assessment to a program’s IT system.

Managing data quality requires establishing processes to ensure that data collected is accurate and used appropriately for evaluative purposes. Inaccurate or misleading data can invalidate evaluation results and negatively impact program performance. Strategies for data quality management include establishing data collection processes that minimize the risk of collecting bad data (e.g., pre-populated fields, testing of data collection forms/tools, clear directions and training for anyone collecting program data), desk-auditing collected data for perceived anomalies, onsite verification, and other strategies.

Energy modeling, or simulation, is the practice of using software to model the energy performance of a home or its systems. Modeling is typically used by contractors or programs to estimate the energy savings that a customer could expect from completing various energy efficiency improvements.

To predict energy savings, program administrators select among different approaches (e.g., modeled savings, deemed savings, hybrid methods) that use information about the quantity and type of building improvements and engineering estimates of expected energy savings. Energy savings estimates may differ from actual energy use because of weather, home size, number of occupants, occupant behavior, installation quality, or other factors.

Impact evaluations are the traditional approach to verifying program energy savings because they measure actual reductions in consumption by a residential energy efficiency program’s participants. Formal impact evaluations seek to isolate the attributable impacts of a program’s efforts by adjusting for weather differences and independent forces by comparison with a control group.

Incentives are tools to motivate potential or current customers or stakeholders to take a prescribed action by lowering risk and decreasing upfront costs. The incentive can be financial (e.g., rebates, limited-time offers, special interest rates) or non-financial (e.g., public recognition, prizes, awards).

Information Technology (IT) systems help program administrators efficiently and effectively track energy upgrade activities, workflow, and data involving customers, contractors, utility and financial partners, and other stakeholders. Program managers need to consider the pros and cons of buying or developing a data collection solution that will meet their specific needs for aggregating, analyzing, storing, and visualizing data for program management, as well as a source of data for evaluations.

The financial industry includes a range of lending institution types. These include community banks, credit unions, non-bank finance companies (leasing company or specialized financial institutions), national energy efficiency lenders, community development financial institutions, utility-partnering lenders, and state-chartered (state-level) bond authorities. Each lender type will have its own goals and unique perspectives with respect to financing residential energy efficiency upgrades.

Neighborhood sweeps are geographically targeted campaigns to reach a specific audience in an identified community over a defined period of time.

Energy efficiency programs provide identifiable benefits beyond energy savings, such as health and safety improvements, job creation, economic development, avoided emissions, and water savings. Quantifying these non-energy benefits may help program administrators demonstrate progress toward stated program and/or policy goals, or increase general awareness and support for program activities.

Pilot projects allow programs to gain direct experience in their markets, while testing and refining program design and processes before full-scale launch.

A process evaluation systematically assesses an energy efficiency program in order to document its operations and identify improvements that would enhance its efficiency or effectiveness, while maintaining high levels of participant satisfaction. A process evaluation may be accomplished in-house or performed by an independent third-party.

Process flow diagrams illustrate key steps, decision points, and interaction points between programs, contractors, and partners from home energy upgrade project inception to completion. They are an important tool for ensuring effective coordination at critical points in the assessment and upgrade process, and identifying opportunities to streamline program processes.

Program dashboards are tracking tools that summarize metrics for monitoring progress toward meeting program goals, objectives, and efficient program processes. For many programs, they are an important tool for assessing and improving programs over time and communicating results to partners and stakeholders.

Quality assurance ensures that upgrades meets agreed upon technical standards established by programs, contractors, and/or other partners. Strategies range from requiring certification of contractors, through processes to ensure the technical quality of installed improvements, to soliciting customer feedback once projects have been fully completed. Quality assurance protects homeowners by providing an independent review of the upgrade work performed by contractors to ensure that it meets programs' technical standards. Quality assurance also protects the reputation of a program.

A request for proposals (RFP) is often necessary to engage the services of a program implementation partner or third party evaluator. A RFP should have a well-defined scope of work and clear description of how proposals will be evaluated.

Funding for organizational and program activities can come from a variety of sources including, but not limited to, ratepayer funds collected by utilities, grants (federal, state, foundation, etc.), and income from services provided to program participants or contractors. In many cases, individual revenue streams will have specific requirements on how the funding may be spent as well as specific reporting requirements.

Public, private, and non-profit organizations often seek to work in partnership with investor-owned and municipal utilities to provide energy efficiency services. Utilities may already offer energy efficiency services that other organizations can enhance or promote, and utilities typically have access to energy consumption data that helps track program success.

Last Updated: 04/01/2016