To support your external evaluation activities, you will need to develop resources to collect, transfer, store, and review data. This may include information about building upgrades, energy usage, program costs, or participant comments – whatever key performance metrics you have identified which will help to evaluate your program’s success. To obtain or transfer some of this information, you may need formal data release agreements from utilities, participants, or others.
The success of your program evaluation will depend on the careful and timely development of these data collection resources. Like your evaluation plans, the processes and tools necessary to support effective data collection for program evaluation should be developed as you are designing your program, and all tools should be tested and implemented before program launch.
This handbook discusses the steps you should take and questions you should ask as you develop the resources necessary to implement successful internal and external evaluations of your program:
- Identify the types of data you need and how you will collect the data
- Identify the processes and tools for collecting and assessing program data
- Test your tools and processes before program launch.
Step by Step
Data will be the primary resource in your evaluation. In order for evaluations to be effective, the data need to be collected, transferred, and stored in a way that ensures accuracy and provides context to what you want to evaluate.
Conduct and manage evaluation and data collection activities
Your evaluation plan should specify the most important data to collect. Depending on your evaluation goals, consider where you are likely to get the most accurate data to assess your key performance indicators and if you are meeting your program’s goals and objectives:
- Impact evaluations determine program-specific induced effects, such as reductions in energy use, demand, and non-energy benefits. Because they focus on energy usage before and after building energy efficiency improvements are completed, customers and utilities will be the primary sources of data. These data may also be used to evaluate the program’s impact on non-energy goals, such as reductions in water usage or carbon emissions.
- Process evaluations focus on the efficiency and effectiveness of program operations and depend more heavily on interviews and metrics from program staff, subcontractors, and customers. You will need to give evaluators the contact information for appropriate staff, subcontractors, and customers.
For guidance on defining key performance indicators, and what data to collect to measure them, consider using the Guide for Benchmarking Residential Energy Efficiency Program Progress With Examples. It provides an inventory of recommended Residential Program Progress Metrics, describes approaches for using them effectively, and gives examples of peer benchmarks from the Better Buildings Neighborhood Program for comparison. In addition, the Guide lays out a step-by-step action plan for developing an internal benchmarking plan for your residential program.
Energy Consumption Data Access
The amount of electricity, fuel, and water a building uses is recorded using meters. Utilities collect this information and use it to bill their customers. For energy (i.e., electric, natural gas) and water consumption data, work with your utility and come to an agreement on the forms and processes for release of information. The utility may already have a form that must be used per its legal department or the utility commission may require a specific form.
For more information on how utilities, policymakers, building managers, and community stakeholders can improve access to energy usage data while working towards the goal of improving efficiency in your community, see ACEEE’s Best Practices for Working with Utilities to Improve Access to Energy Usage Data Toolkit.
Helpful Tips for Collecting Energy Usage Data
Connecticut’s Neighbor to Neighbor (N2N) Energy Challenge successfully established a partnership with its utility and suggests the following tips for a successful relationship:
- Host a kickoff meeting to get on the same page about expectations. Within the first few weeks of their partnership, N2N held a kickoff meeting with Northeast Utilities (NU) and began discussing the data needed. In addition to the individual household usage data that the Better Buildings Neighborhood Program requires, N2N wanted to collect community-wide aggregate usage data, as well as historical community-wide data on participation in state efficiency programs in N2N’s towns. This kickoff meeting helped N2N to ensure that its data needs were within the realms of what NU was able to share.
- Allow ample time for negotiations and technical processing. The process to access data can take time and patience. For Connecticut, the process took six months of negotiation and another two months of development. In addition to figuring out how to release the data, which included many legal issues, both parties had to work with their IT security departments to allow electronic access to the data. In the end, N2N’s patience paid off as NU worked with its IT staff to provide regular data in a meaningful format for N2N.
- Be flexible. On the first Monday of the month, N2N gives NU utility account numbers for residents who have agreed to share their data. NU provides usage data for these homeowners to N2N by the last day of the month. Although N2N had hoped for more frequent access to utility data, N2N recognizes the need to work within a utility partner’s technical and operating environment.
- In the agreement with the utility, explain where the data will be shared. To ensure that further negotiations were not needed, N2N’s agreement with NU included the utility data uses, which included sharing data with the Better Buildings Neighborhood Program. N2N requires a non-disclosure agreement with any approved partner accessing utility usage data.
Despite the involved process, gaining access to data was worthwhile for N2N. N2N created a strong relationship with NU at the beginning of the process, so the NU team was helpful, professional, and willing to invest the time necessary to help N2N collect utility data.
Source: Better Buildings Neighborhood Program Final Report, Connecticut’s Neighbor to Neighbor (N2N) Energy Challenge, 2014.
Utility customer energy usage requests should identify the utility contact and program contact responsible for data reporting and the specific data requested (i.e., meters/accounts/homes for which you need data). Include in the request the period for which customer usage information will be provided. For example, monthly energy usage for 12 to 15 months pre- and post-building upgrade completion is often necessary for impact evaluation billing analysis.
Avoid confusion from the start of the program by also specifying in the agreement:
- The format for data collection, by using a data template such as the Utility Customer Energy Usage Data Template developed by the U.S. Department of Energy
- How energy usage information will be transferred (e.g., electronically, paper, phone)
- How data will be stored
- How often data will be shared
- Who will have access to these data
- What data security training is required
Identify the supporting documentation you will need to request building energy consumption data. For example, will separate data requests need to be signed by each participant? Can existing procedures be used, such as including the data release as part of program enrollment forms or as part of the energy assessment?
If utilities provide paper copies of energy usage bills, convert the data to machine readable text and save it in an electronic format.
Tips for Data Accuracy
- Make sure that energy usage data are for the building that was upgraded, rather than the customer.
- Identify energy usage using the meter number, building address, account number, or customer name.
- Note that meter number is more reliable than customer name.
- A customer may have more than one building or account; also, if your customer moves six months after participating in your program and your evaluator wants to examine the 12-month period after program participation, you do not want your data contaminated by six months of usage data at a different residence.
Bulk Fuel Data
Programs that serve homes using bulk fuels (i.e., oil, propane, and wood) for water or space heating need to determine how to capture pre- and post-project non-utility fuel consumption.
Some fuel providers may be unable or unwilling to provide you with sales records showing bulk fuel purchases, even with the customer permission. Providing sales records may be difficult and time-consuming if the fuel providers’ systems were not designed for this purpose. You should still explore this avenue as a first, best option, but also recognize that there may be dozens of providers or more, ranging from the very large to the very small.
Without data from fuel providers, you will need to capture fuel usage information individually from the customer, whether from their records or their estimates. In either case, you should be prepared to receive data in various formats and units and plan to make the necessary translation to a common unit of measure.
Overcoming Difficulties Collecting Bulk Fuel Data (e.g., fuel oil, propane)
Community Power Works (CPW) in Seattle, Washington reached out to a sample of oil providers to gauge interest in providing fuel consumption data and discovered through this outreach that data collection and verification would cost the program upwards of $50,000. Instead CPW, used Earth Advantage’s Energy Performance Score modeling software to estimate energy use.
In Their Own Words: Energize New York Overcomes Difficulities Collecting Non-Utility Fuel Data
Source: U.S. Department of Energy, 2012.
Customer, Contractor, and Partner Feedback
In addition to quantitative information about program accomplishments and customers, you will need to collect and use qualitative information about your program, including those described below.
- Customer feedback. Feedback from customers can give you information about customer satisfaction (with the program overall as well as with specific program components or partners). Customer feedback can be gathered via customer surveys and from contractors who can provide insights from their direct contact with customers.
The following sample customer surveys and survey questions were developed by the Better Buildings Neighborhood Program:
- Sample email survey template for successful program participants
- Research questions and survey questions for successful, drop-out and screened-out program participants
- Sample phone survey template for energy efficiency program drop-outs
- Sample phone survey for applicants who have been screened out from participating in and energy efficiency program.
For more information on customer surveys see the Program Design & Customer Experience – Develop Evaluation Plans handbook.
- Contractor feedback. To get feedback on program processes and strategies, many programs establish ongoing interactions via surveys and regular meetings with contractors. More information on contractor feedback can be found in the Contractor Engagement & Workforce Development - Assess & Improve Processes handbook.
- Partner feedback. Your partners can be an excellent source of feedback about your program based on their interactions with you as well as their ongoing interactions with customers, contractors, and others.
Customer Income Data
If you have income requirements for your program, your evaluators may need to verify the processes by which you confirmed that household income met the requirements of your program. Be sure that these processes are well documented for each participant in the program. You should also explore collaborating with organizations that already serve your target population. For example, state housing finance agencies may have programs that capture income information for homeowners in the target population for your program. Rather than have the homeowner provide income verification to two different organizations, you could explore whether the housing finance agency can verify income qualification on their behalf. In this instance, you will likely need to sign a confidentiality agreement with a partner organization so that they (and your program’s customers) are assured that you will use the information only for agreed-upon purposes.
Some data you collect may be confidential, such as energy consumption, income, household demographics, or survey results. You must ensure that all relevant parties—both staff and contractors—are meeting customers’ needs for confidentiality. To do so, you will need to develop and enforce policies and protocols for staff and subcontractors with access to confidential client data.
In addition to finding the data necessary to measure your program’s success, you need to determine how those data will be collected, transferred, and stored. This step is more than simply acquiring a software tool—it requires careful planning and development of data collection processes, protocols, and supporting tools for capturing the necessary metrics. Procedures and tools for data collection, processing, and analysis can span a wide range of options from simple to elaborate, so think about what will meet your needs.
To begin this process, answer the following questions:
- What tools are already in place that you can use to meet evaluation data collection needs?
- If your organization runs similar programs, it may have systems in place for collecting data. For example, low-income weatherization programs often have established data collection forms, data release agreements, and program management systems that might be modified for your program. They can also share what they have learned from experience.
- The Better Buildings Residential Network holds periodic peer exchange calls to discuss program topics, including evaluation and data collection. Learn more about how to become a member.
- What tools can be adapted to meet evaluation data collection needs?
- If an existing system is not suitable as is but can be adapted, it may be a lower-cost option than procuring a new system.
- While there are already-developed systems designed to support energy efficiency program data collection, you will likely need to invest in customization to meet your program goals and metrics. The costs, in both time and money, need to be incorporated into your decision-making process.
- Do you need outside assistance?
- You may not need outside assistance if your existing systems and tools meet the data collection needs you have identified.
- If a software data collection system from an outside vendor needs to be modified for your program, you will need to consider the time and cost for adaptation.
- If no existing tools can be used or adapted, you may need to procure services to build a software tool.
Important Standards and Guides for Data Collection and Measurement
Residential energy efficiency programs and the resources dedicated to them have increased dramatically in recent years, as have the tools that enable more precise data gathering and measurement calculations. This has prompted efforts on the part of program administrators and stakeholders to standardize elements that simplify data collection, transfer, and calculations. The following standards and guides can help you increase the effectiveness of program evaluations and benchmarking activities by aligning with common data collection and measurement protocols.
- The U.S. DOE Building Energy Data Exchange Specification (BEDES, pronounced "beads") supports analysis of the measured energy performance of commercial, multifamily, and residential buildings, by providing common data terms, definitions, and field formats for building characteristics, efficiency measures, and energy use.
- The U.S. DOE Standard Energy Efficiency Data Platform (SEED) is an open source software application that helps organizations manage data on the energy performance of large groups of buildings. Users can combine data from multiple sources, clean and validate it, and share the information with others. The software application provides an easy, flexible, and cost-effective method to improve the quality and availability of data in order to help demonstrate the economic and environmental benefits of energy efficiency, to implement programs, and to target investment activity. The SEED software includes a common, BEDES-compliant database structure that enables users to import data about the same group of buildings from multiple sources, conduct analysis, and report out the information. The SEED code is open source, which means that anyone can add to or modify the software. This prevents vendor lock-in and allows flexibility to add and modify features to collect new information and utilize data in new ways.
- Home Performance Extensible Markup Language (HPXML) is a data transfer protocol. It was developed to simplify electronic data transfer between any party involved in a home performance program, including contractors, program administrators, utilities, and federal agencies. HPXML uses a subset of the BEDES' terms and definitions and organizes them into a schema for electronic data transfer, along with exchange protocols, implementation guides, etc. HPXML can be used by appraisers to support valuation of home energy efficiency improvements. See, the HPXML Implementation Guide to learn how to integrate HPXML into your program’s operations.
- The U.S. DOE Uniform Methods Project provides standard protocols for program administrators to use in calculating energy savings attributed to energy efficiency programs. These protocols are meant to ensure consistency and transparency, as well as strengthen the credibility of energy efficiency program savings.
- The SEE Action EM&V Guide provides a wealth of valuable information on planning and conducting evaluations of energy efficiency programs, including methods for calculating gross and net energy savings, cost-effectiveness, and avoided air pollution emissions.
You will need to design data collection systems and processes to ensure that the data are not only collected but correct, so quality assurance is critical. Inaccurate data will lead to misleading evaluation results and are a key concern not only for third-party evaluations of your program, but for effective program management.
You need to have systems and procedures that match your program’s data collection needs and resources in place from the outset, recognizing that continued adjustments will be necessary as you identify areas for improvement and implement program changes.
- The data collection software can be designed so that some data range checks and completeness reviews can be performed to flag potential data errors.
- You should also implement regular desk audits of reported data (i.e., reviewing reported data for outliers and suspected errors), as people often make mistakes in ways that cannot be anticipated and therefore cannot be checked by the software. Make sure to follow up immediately on potential errors you catch this way.
- Training on data collection forms and systems and clear definitions for the data elements being collected is essential for anyone responsible for collecting and reporting data. It’s also important—and can improve data quality—to give these people a strong sense of connection to program outcomes. Data collection and reporting takes a lot of their time, so make sure they know why their data are important to the program’s success.
Best Practices for Data Quality
- Data requirements for measuring program success should be clearly defined so that staff responsible for data collection understand the types of data that will be collected, the quality of data required, and how to collect the data.
- Use internet-based data collection and tracking systems which include data validation functions to assess data quality.
- Develop and distribute a user manual to all who are responsible for entering and/or validating data. Keep the manual updated with any changes to data collection processes and protocols.
- Automate all regular reports, such as monthly job reporting and financial tracking, as much as possible. Check reports frequently for possible data errors.
- The tracking system should integrate marketing, customer, financing, and individual upgrade level data. This helps to tell the full story as well as highlight potential data errors or quality assurance issues (e.g., measures installed that were not reflected in the audit).
Data collection processes are discussed in more detail in the Program Design – Develop Evaluation Plans handbook.
Data Review and Assessment
Once you have collected data, you need to review and understand the data in a way that allows you to identify what areas might need attention or a change in organizational processes. One way to assess your data is by using an approach known as benchmarking. The two types of benchmarking—internal and external—will:
- Help you understand whether you are over- or underperforming
- Help you determine if you need to refine your vision, mission, and goals to be more realistic
- Allow you to communicate progress or failures
- Enable you to assess where there are opportunities for improvement and develop a process for implementing changes.
Internal benchmarking means you compare your current performance to your own program’s performance in the past. External benchmarking is comparing your program’s performance with a group of organizations similar to yours with which to compare your results. For additional information about benchmarking see the Market Position & Business Model – Assess and Improve Processes handbook.
DOE Guide for Benchmarking Residential Energy Efficiency Program Progress With Examples
The U.S. Department of Energy developed a guide for benchmarking residential energy efficiency program progress. The guide lays out an action plan for developing an internal benchmarking plan for your residential program.
Depending on your current operations, you may not need to implement all of these steps, but consider each of them when developing your own benchmarking plan.
- Step 1. Use Program Goals to Guide Benchmark Planning
- Step 2. Identify Potential Metrics that Measure Your Goals
- Step 3. Determine How You Will Collect the Information
- Step 4. Assess Level of Effort and Finalize Metrics
- Step 5. Put the Process in Place and Get Started!
- Step 6. Share Results Effectively
- Step 7. Consider Benchmarking Against Peer Programs
Before full-scale program implementation, you should thoroughly test your data collection processes and systems. This includes testing your data collection forms, reporting systems, and quality assurance processes. You can do this by recruiting a number of program staff members or subcontractors to implement the data collection and reporting process, using a number of projects to test the systems.
Testing data collection processes and systems with an initial pilot or soft launch of your program will give you a chance to test the forms and reporting effectiveness on actual energy upgrades, not hypothetical scenarios. Even testing forms (particularly electronic data collection forms) and reporting systems with hypothetical data, though, is better than not testing at all.
In Their Own Words: Avoid Contractor Frustration by Testing Data Collection Tools Before Deployment
Source: U.S. Department of Energy, 2012.
Tips for Success
In recent years, hundreds of communities have been working to promote home energy upgrades through programs such as the Better Buildings Neighborhood Program, Home Performance with ENERGY STAR, utility-sponsored programs, and others. The following tips present the top lessons these programs want to share related to this handbook. This list is not exhaustive.
Many Better Buildings Neighborhood Program partners found that setting up their information technology (IT) systems early in the program design stage ensured that data terms and data entry procedures were consistently applied by all system users. Reaching agreement with stakeholders (e.g., contractors, lenders, marketing partners, evaluators, program staff) on what data the data system would collect, known as system requirements, and how the collected data would be used to evaluate the program helped programs ensure that the data collected was complete. Programs have also found that they receive data of the quality needed for graphs and cost-effectiveness calculations when stakeholders agree up front that the data will be used for these purposes and not just to track energy savings and expenditures.
- Be SMART Maryland found that transitioning from spreadsheet-based data collection system to a customized energy IT system was crucial to administering a multifaceted energy efficiency program with rigorous data collection requirements. Investing in their system while they were still designing their program allowed Be SMART to smoothly integrate the system into the program’s operations and to ensure quality data collection and integrity over time. Be SMART also found that while spreadsheets were useful tools in collecting data, their use in analyzing data and generating reports was limited, since the program had to go through a time-consuming consolidation process to combine data from different sources and spreadsheets.
- In Boulder County, Colorado, EnergySmart found that it took between four to six months for a database developer and coding consultant to fully develop and test the data system because of its high level of complexity and the customization required. The program also found that having actual users test the system with real inputs and real reporting requirements helped ensure better data quality and user-friendliness. In addition, EnergySmart found that before beginning database development, it was important to reach agreement among stakeholders on what reporting will be expected, and design the database to facilitate building and exporting the reports. For EnergySmart, it was important to set expectations with report recipients about the IT system’s reporting capabilities early on in the process, so recipients did not expect reports that the system was unable to produce.
Paper-based or spreadsheet-based information collection processes can be low cost to develop and easy to roll-out, but more often than not, they become cumbersome to aggregate and store the data from many sources. Many Better Buildings Neighborhood Program partners found that investing in information and communications technologies (ICT) eased program implementation and was well worth the effort because they were able to regularly monitoring progress and automate what would otherwise be time-intensive, manual processes. For more information on the future of ICT, see ACEEE’s How Information and Communications Technologies Will Change the Evaluation, Measurement, and Verification of Energy Efficiency Programs.
- Garfield Clean Energy in Garfield County, Colorado, at first used a series of Excel spreadsheets and hardcopy file folders to track participants, their energy upgrade measures, and resulting energy savings. As the number of participants reached into the hundreds, the program realized that spreadsheets did not offer the level of sophisticated searching and reporting that was needed to analyze the results of their work. They explored several online customer relationship management systems and contracted with a third-party developer to customize their selected system so that it could track building and energy data, energy upgrades, contractors, dollars spent, rebates awarded, and deemed energy savings. The customization and data entry work, which took several months to complete, enabled Garfield Clean Energy to create detailed reports based on a wide variety of reporting parameters, and to better analyze the effectiveness of program activities.
- When Enhabit, formerly Clean Energy Works Oregon, scaled up their pilot program to the entire city of Portland, it was clear to them that an IT solution was needed to meet the demands of funding agencies, media requests, and good project management from the customer perspective. Enhabit worked with software company EnergySavvy to develop a unified service delivery platform to manage the home energy upgrade process from application to completion. The software platform provides a web-based interface between homeowners, contractors, and lenders, enabling each party to document progress through the Enhabit program. The platform also streamlined data collection and analysis.
- In Boulder, Colorado, EnergySmart used spreadsheets to manage data for its predecessor energy efficiency programs. As the program expanded under the Better Buildings Neighborhood Program, it became clear that EnergySmart needed to pursue a more user-friendly, real-time, cloud-based IT system for tracking customers through the implementation process. They selected a system to allow for tracking of many metrics in a much more consistent, accurate, and organized fashion than the previously used spreadsheet. The system can be accessed in the field by EnergySmart Energy Advisors using iPads or tablets to enter basic customer information, building baseline information, assessment findings for upgrade opportunities, completed upgrades with associated energy and cost savings, rebates and financing received, and the supporting documentation. The collected data is compiled for reporting to various stakeholders, including the U.S. Department of Energy, county commissioners, and city staff and leaders. The ability of Energy Advisors to access the system in the field allows for much greater efficiency and accuracy than the static logging of data upon returning to the office.
Many Better Buildings Neighborhood Program partners found that it was critically important to use compatible formats for data sharing and reporting with partners. Aligning data formats and collection plans with national data formats (e.g., Home Performance XML schema (HPXML), Standard Energy Efficiency Data platform (SEED), Building Energy Data Exchange (BEDES)) ensured compatibility for aggregation and reporting.
- For Arizona Public Service’s (APS) Home Performance with ENERGY STAR® Program, a lack of transparency and access to data meant it took hours each month to compile progress reports. Coordination with trade allies was difficult for similar reasons–both the utility and its contractors lacked visibility into project status and task assignment, as well as the ability to identify program bottlenecks, which impacted APS customer service. Program delivery metrics, from administrative overhead to customer and trade ally satisfaction, were lower than expected. APS then began the search for a more dynamic software platform to engage customers, track and manage projects, empower trade allies, and analyze and report results. The program needed HPXML, an open standard that enables different software tools to easily share home performance data. The new HPXML-compliant platform, EnergySavvy’s Optix Manage, resulted in higher cost effectiveness and greater satisfaction for the program, including 50% less administrative time to review and approve projects, a 66% reduction in data processing time for APS reporting, 31% less contractor administrative time to submit projects, and a three-fold increase in trade ally satisfaction. HPXML also had the added benefit that contractors can choose their own modeling software.
- The New York State Energy Research & Development Authority (NYSERDA) heard from home performance contractors and other stakeholders that a more streamlined data collection process was needed to reduce the paperwork burden and time spent on a per project basis. In response, the program launched the NY Home Performance Portal in July 2013. This web-based interface made it easier for customers to choose and apply for the home performance program and made the application process for a home energy assessment clear, fast, and simple. In 2015, NYSERDA further refined their data collection process and began processing of all projects in a web-enabled interface designed to facilitate program coordination. This new platform allowed NYSERDA to automate project approvals for 85-90% of projects. In addition, the platform supported HPXML which facilitates data sharing among multiple New York programs, thereby reducing the administrative burden for contractors participating in multiple programs. It allowed NYSERDA to automate the work scope approval process through validation of standardized data. An additional benefit of HPXML for NYSERDA was creating an open modeling software market.
- Massachusetts Department of Energy Resources (MassDOER) provides statewide oversight to energy efficient programs administered by utilities under the Mass Save brand. Originally, contractors from Conservation Services Group, Inc. and Honeywell International Inc. used audit software customized for the program in their home energy assessments. When MassSave piloted the Home MPG program, contractors were also required to generate an Energy Performance Scorecard for each home. The existing audit software, however, did not have this capability. To address this problem, software developers added the Energy Performance Scorecard capability, so the contractors could use the same software to record the audit results and generate the energy performance scorecard. Despite implementation delays, this solution allows the use of the Energy Performance Scorecards to potentially expand to statewide.
Many Better Buildings Neighborhood Program partners found that program dashboards—regularly updated, easily accessed, summary reports of key metrics—helped them identify problems and monitor program progress toward their goals. Depending on the program’s goals and needs, dashboards included metrics such as energy savings, energy upgrades in progress and completed, customer satisfaction, jobs created, and cost of service delivery.
- Connecticut's Neighbor to Neighbor Energy Challenge (N2N) customized their Salesforce system to display dashboards of key project data, such as contractor performance and the marketing and outreach activities of staff and volunteers. Contractors and program staff entered all the data into Salesforce. Program administrators reviewed the data on a weekly basis to ensure data quality and resolved any data quality issues with contractors. Program administrators met with outreach and contractor staff to discuss the dashboard findings and any necessary process refinements and course corrections. The program also published monthly dashboards to share data such as number of outreach events, number of leads generated, and number of upgrades completed, with state policymakers and other program partners.
- Massachusetts Department of Energy Resources (MassDOER) developed a monthly dashboard that tracked the number of homes assessed, the number of home energy upgrades, and signed efficiency energy upgrade contracts. The dashboard included cumulative data and data for the past month. MassDOER used the dashboard to track the gross number of audits and upgrades, and to understand what work was “in the pipeline.” This helped MassDOER determine whether they were on track to meet project goals for assessments and upgrades. MassDOER had confidence in the data because staff reviewed the updated dashboards each month, flagged inconsistencies, and confirmed the numbers with the contractors who supplied the data, making corrections when it was necessary.
Many Better Buildings Neighborhood Program partners found that it is important to get buy-in from program staff and contractors on the importance of data integrity to the program mission and then to invest time to develop materials and train everyone who has a role in data collection and analysis. In this way, programs administrators can ensure that staff understand the what, how, and why of data collection and analysis requirements.
- In the Maryland Department of Housing & Community Development’s (DHCD) commercial, multifamily and residential programs - collectively known as Be SMART Maryland, discrepancies arose in the variety of ways contractors determined energy savings and the type of energy modeling software used. Contractors utilized diverse auditing software which in turn used different factors and algorithms to determining overall energy savings. At times, their upgrade reports submitted to the program omitted many input fields used to determine the energy cost savings and did not include units of measure (therms, kWh, etc.). Be SMART Maryland developed a standard methodology for estimating energy savings by requiring certain information fields such as estimated energy savings percent and having DHCD quality assurance inspectors (who are Building Performance Institute, Inc. certified and trained in energy auditing) review energy audits reports for reasonableness. For the multifamily energy efficiency programs, the program developed a standard energy audit guide, and energy auditors were required to follow one of several DOE approved audit approaches.
- The Missouri Agricultural Energy Saving Team —A Revolutionary Opportunity (MAESTRO) program faced a challenge concerning the level of specificity in auditors’ reports and the precision used in determining audit results. In many of the most rural areas of the state, auditors were simply unaccustomed to the level of precision required by the program. To address the problem with those auditors, the program identified the utility company supplying energy to the audited homes, identified their utility rates at the time of the audit, and then applied these data to a data-driven model established by auditors who operated at the required level of quantifiable data.
- Seattle’s Community Power Works (CPW) program provided contractors with multiple trainings and a well-staffed hotline for communication and assistance throughout the grant period. Training began with a program overview session for new participating contractors focused on program logistics that informed contractors of the program’s intent, rebate structures, and expectations. Other trainings covered basic safety, quality oversight, and use of their mandatory reporting database, which the program used to track projects and workforce development. To help contractors improve their performance, program staff used survey results to provide feedback to contractors on quality assurance, customer satisfaction, and compliance with program requirements.
Though potentially challenging, establishing relationships for sharing energy consumption data is critical for evaluating program impact on energy and cost savings. Many Better Buildings Neighborhood Program partners found success by approaching utilities during the program planning phase, or at least several months in advance of when they planned to start collecting data, to outline shared goals, assets, tools, needs and constraints. Clear and concise data requests helped speed up utilities’ response times for providing the data and alleviated utility concerns and questions regarding data needs.
- Energize Phoenix formed a partnership with the local electric utility, Arizona Public Service (APS), while designing the program and coordinated with them throughout program development. Energize Phoenix found that understanding Arizona Public Service’s concerns and challenges related to data sharing was a key ingredient in forging a successful partnership, as was instituting a formal agreement to clarify roles and responsibilities.
- Southeast Energy Efficiency Alliance (SEEA) found that not all of their programs were successful in obtaining utility bill data. Common obstacles included that the utility did not have the technology infrastructure to easily export the information, would only release data for a fee (based on how many records were pulled), or simply did not have the time or resources to provide the information even if the program had a signed client release form from the homeowner. Among SEEA's programs, those that were most successful in obtaining utility billing information–including NOLA WISE in New Orleans, Louisiana; Local Energy Alliance Program (LEAP) in Charlottesville, Virginia; Atlanta SHINE in Atlanta, Georgia; and DecaturWISE in Decatur, Georgia - consulted with the utilities to determine what information the program needed to include in the client release form. Additionally, some programs developed a written memorandum of understanding with the utility specifying data collection and transfer roles and responsibilities. SEEA programs also found it best to make data requests to utilities on a quarterly basis to minimize the burden on the utility as many utilities do not have staff dedicated to data exporting. Some programs received data more frequently, but in these situations the utility had the means to easily pull and export data.
- When local utilities Philadelphia Gas Works (PGW) and Philadelphia Electric Company (PECO) shared customers’ energy usage data with EnergyWorks, all parties made sure that the proper data sharing requirements were observed and all parties signed the necessary forms. Philadelphia EnergyWorks built its customer data release approval language into the program’s loan application form to minimize the number of additional forms that a customer or contractor would need to handle.
- EnergySmart in Eagle County, Colorado, successfully developed partnerships with utilities during and after the Better Buildings Neighborhood Program grant period, but in hindsight found it would have been more beneficial to engage utilities prior to submitting the original DOE grant application. By not fully engaging utilities up front, EnergySmart created the following environment where the utilities are only partially included in the program and retained similar or redundant in-house services. As EnergySmart continued forward, they were able to gain the trust of the utility by offering help, data, and information. EnergySmart also shared their results with the utility’s management and board of directors. Through this gained trust, utilities were more willing to share data.
The following resources provide topical information related to this handbook. The resources include a variety of information ranging from case studies and examples to presentations and webcasts. The U.S. Department of Energy does not endorse these materials.
Delaware Sustainable Energy Utility Request for Qualifications for Home Performance with ENERGY STAR with Complimentary Home Energy Check Up Program Implementation Contractor
This RFQ from Delaware Sustainable Energy Utility (DESEU) seeks an implementation firm that can deliver the Home Performance with ENERGY STAR (HPwES) Program cost-effectively for a three-year period.
Delaware Sustainable Energy Utility Home Performance with ENERGY STAR Programs: Benchmarking and Emerging Trends
This report consists of a literature review and in-depth interviews with subject matter experts in the Home Performance with ENERGY STAR (HPwES) program area. The goal was to compare Delaware Sustainable Energy Utility (DESEU) HPwES Programs with peer-programs across the United States. The report also identifies key metrics and emerging trends regarding program design.
Delaware Sustainable Energy Utility Home Performance with ENERGY STAR: Program Evaluation & Program Re-Bid Recommendations
Delaware Sustainable Energy Utility (DESEU) completed a process evaluation to assess the overall effectiveness of program operations. Evaluation activities included reviewing program materials, assessing the program flow, conducting in-depth interviews with program staff and implementers, and conducting surveys with contractors and customers. This report summarizes the key findings and recommendations from these process evaluation activities.
This presentation describes how programs have leveraged data to increase program energy savings, with a spotlight on advanced and real-time monitoring and verification (M&V 2.0), contractor scorecards, and intelligent quality assurance (QA) and monitoring.
This is a recording of a webinar from August 2015. Home Performance with ENERGY STAR hosted a panel on HPXML; the value it can bring to businesses and implementation methods. Interested organizations can use this resource to learn more about HPXML and its potential benefits.
Home Performance (HP) XML is transforming the way home energy upgrade programs collect and transfer information from one software system to another, leading to improved contractor satisfaction, lower administrative costs, and technological advancements in the home performance industry. This presentation provides an overview of HPXML and its benefits, and discuss how the data standard is facilitating technological and process improvements among home energy upgrade programs and software developers in the United States.
This webinar series is intended for state officials starting or expanding their EM&V methods for a wide range of efficiency activities including utility customer-funded programs, building energy codes, appliance and equipment standards, energy savings performance contracting, and efficiency programs that support pollution reduction goals or regulations.
The ultimate objective of the protocol is to develop a system that can be used to guide the production of readily usable dataset that can leverage project data from future meter-based measurement and evaluation studies, or metering studies to develop end-use load shapes. The protocol includes a "NEEP Residential Data Collection Protocol Report" template.
Form used by the Colorado Public Utilities Commission for consent to disclose utility customer data.
Homeowner survey created by the utility to inform their whole home upgrade program.
This sample email survey template, created by the Better Buildings Neighborhood Program, was designed for programs to develop their own survey of successful program participants in order to assess customer experience.
This sample phone survey template, created by the Better Buildings Neighborhood Program, was designed for programs to use with applicants who have been screened out from participating in a program.
This sample phone survey template for program drop-outs, created by the Better Buildings Neighborhood Program, was designed for programs to find out why applicants that applied to participate in a program ultimately dropped out.
Sample phone survey template for program contractors.
This template, used by Clean Energy Works Oregon (now Enhabit), standardizes a number of forms that contractors fill out for the program.
The Energy Data Accelerator Toolkit is a collection of resources featured in the Better Buildings Solution Center that will enable other utilities and communities to learn and benefit from the work of the Accelerator. It describes the best practices that enabled cities, utilities, and other stakeholders to overcome whole-building data access barriers.
List of building energy software packages, some of which are available for free or a small fee.
The State and Local Energy Efficiency Action Network (SEE Action) Evaluation, Measurement, and Verification (EM&V) Resource Portal
The State and Local Energy Efficiency Action Network (SEE Action) Evaluation, Measurement, and Verification (EM&V) Resource Portal serves as an EM&V resource one-stop shop for energy efficiency program administrators and project managers. The resources focus on tools and approaches that can be applied nationwide, address EM&V consistency, and are recognized by the industry.
The Building Energy Data Exchange Specification (BEDES, pronounced "beads" or /bi:ds/) is designed to support analysis of the measured energy performance of commercial, multifamily, and residential buildings, by providing a common data format, definitions, and an exchange protocol for building characteristics, efficiency measures, and energy use.
Home performance extensible markup language (HPXML) is a national Building Performance Institute Data Dictionary and Standard Transfer Protocol created to reduce transactional costs associated with exchanging information between market actors. This website provides resources to help stakeholders implement HPXML and stay updated on its development.
The Standard Energy Efficiency Data (SEED)™ Platform is a software application that helps organizations easily manage data on the energy performance of large groups of buildings. Users can combine data from multiple sources, clean and validate it, and share the information with others. The software application provides an easy, flexible, and cost-effective method to improve the quality and availability of data to help demonstrate the economic and environmental benefits of energy efficiency, to implement programs, and to target investment activity.
The energy efficiency reporting tool for public power utilities is an Excel-based template is designed to produce consistent, useful metrics on program investments and performance for small to medium-sized administrators of public power efficiency programs.
Partnering with Utilities Part 1 -- Successful Partnerships and Lessons from the Field
Partnering with Utilities Part 2-Topics for Local Governments-Creating Successful Partnerships with Utilities to Deliver Energy Efficiency Programs
This webcast focused on advanced topics for local government-utility partnerships, with presentations from local governments and their partnering utilities that have well-developed, multi-year relationships and programs.
Door-to-Door Outreach and Tracking Impacts
This webcast discusses door-to-door campaigns and how to track the impacts of these campaigns.
Guidelines for Retrieving Customer Usage Data from Utilities
Leveraging EPA's Portfolio Manager in Benchmarking and Disclosure Policy
Presentation on the Energy Efficiency Reporting Tool for Public Power Utilities
This presentation discusses the energy efficiency reporting tool for public power utilities. The tool is an Excel-based template is designed to produce consistent, useful metrics on program investments and performance for small to medium-sized administrators of public power efficiency programs.
Better Buildings Neighborhood Program Evaluation Report: Volume 1. Evaluation of the Better Buildings Neighborhood Program (Final Synthesis Report)
Volume 1 of the Better Buildings Neighborhood Program Evaluation Report provides findings from a comprehensive impact, process, and market effects evaluation of the program period, spanning from September 2010 through August 2013.
Better Buildings Neighborhood Program Evaluation Report: Volume 6: Spotlight on Key Program Strategies From the Better Buildings Neighborhood Program
Volume 6 of the Better Buildings Neighborhood Program Evaluation Report provides findings from a comprehensive impact, process, and market effects evaluation of the program period, spanning from September 2010 through August 2013. This volume includes case studies that describe successful strategies that programs used during the evaluation period.