Evaluation of your financing activities is critical to ensuring that you meet your program’s goals and objectives. The collection and evaluation of data on your financing activities will help you identify the strategies and processes that are working and those that could use improvement to help your program achieve success.
Because your financing activities are part of a larger effort by your program to promote home energy upgrades, you will need to develop one overall process evaluation plan that addresses all aspects of your program (e.g., Financing, Program Design & Customer Experience, Marketing & Outreach, Contractor Engagement & Workforce Development. The metrics and measurement strategies for evaluating your financing activities will feed into this overall plan.
Planning the evaluation of your financing activities involves four key steps:
- Establish metrics based on your goals and objectives
- Design the measurement strategies
- Establish how data will be transferred, stored, and managed
- Design the process and the schedule for data review and assessment.
Recall that, ideally, financing will allow:
- More homeowners in your community to proceed with home energy upgrades
- Homeowners to acquire deeper, more comprehensive upgrades.
Your evaluation will assess the extent to which your financing activities help you to achieve these, and other, goals you have established, and the steps you can take to enhance success. The financing metrics you track and the feedback you collect from your evaluation will also be inputs into the continuous improvement process of your program. Metrics (e.g., rate of loan uptake among your target customers) will help you identify whether you need to adjust your financing process.
Step by Step
After making design decisions and developing an implementation plan for your financing activities, your next task is to develop an evaluation plan that will allow you to determine progress toward your program’s goals and objectives.
Remember that financing is part of a larger effort by your program to promote home energy upgrades which involves activities related to marketing and outreach, contractor engagement and workforce development, etc. The evaluation decisions you make regarding financing will feed into one overall process evaluation plan for your program.
The following describes the key evaluation planning steps for financing.
Develop a financing process evaluation plan and metrics
Data that you collect will allow you to measure progress towards meeting your program’s financing goals and objectives and help to improve overall program performance.
For each of your financing objectives, develop a set of metrics that will allow you to measure progress toward meeting that objective. For each metric, you will also want to determine how and when you will measure it. The following table contains examples of financing goals, objectives, and metrics.
Source: U.S. Department of Energy, 2014.
For each metric you identify, make sure to define the data elements to be collected. These may include:
- A qualitative or quantitative response
- The units of measurement.
Each of your metrics and the corresponding data elements can be compiled into a data dictionary for reference by you and your program’s partners.
Common Metrics for Financing Activities
The following are common metrics that you may decide to track. The specific metrics you decide to track will depend on the financing goals and objectives you set for your program.
- Number of loan applications received per time period
- Percent of loan applications approved and rejected
- Reasons for loans being rejected
- Percentage of early payment defaults (e.g., defaults in the first three months)
- Loan performance delinquencies (e.g., 30, 60, and 90 day) and trends (increasing, decreasing, etc.)
- Annual and cumulative default rates, reason for defaults, and trends
- Prepayment rate (loans paying off prior to maturity)
- Severity of loss from defaulted loans (e.g., were any funds collected after the loan was charged-off?)
- Loan and reserve fund balances
- Borrower characteristics (e.g., age, credit score, income, debt ratio)
- Borrower, contractor and lender feedback on loan process
- Elapsed time or date of initiation and completion for the following processes: application, underwriting, origination, and loan funding
Property and Project Characteristics
- Property characteristics (e.g., single-home, apartment, address, age, square footage, energy types)
- Upgrade project characteristics (e.g., list of home energy upgrades financed, cost and estimated savings)
- Number of lending partners
- Contractor characteristics (e.g., name, address, license number, certifications, services offered)
- Number of outreach activities performed (e.g., flyers produced, trainings conducted)
An Initiative to Collect and Analyze Loan Performance and Energy Savings Data from Energy Efficiency Upgrades
The Environmental Defense Fund’s (EDF) Investor Confidence Project, with support from its partners, developed the Open Energy Data Initiative to provide open access to energy and loan performance data and analytics.
This initiative aims to address the lack of data standards and lack of available data on loan performance and energy savings, which capital providers, policy makers, and building owners have cited as limiting their investment in building upgrades.
Lenders rely on rigorous, historical loan performance data (e.g., default rates) in making lending decisions. Since energy efficiency investing is in the early stages of development, historical loan performance data is limited, and what does exist is not aggregated or standardized. Additionally, a wide array of investors, including building owners, energy service companies, insurance providers, and even utilities want to ensure savings are realized and lack the clear data and replicable protocols necessary to accurately underwrite loans.
Currently, contributors to the Open Energy Data Initiative include Enhabit, the Greater Cincinnati Energy Alliance, NYSERDA, and Pennsylvania Treasury. Data includes:
- Personal information about the borrower and property (e.g., credit score, debt-to-income ratio, city, state, building type, year building constructed)
- Loan information (e.g., amount, interest rate, term, loan product, charge off amount, loan capital source)
- Project information (e.g., project work done, contractor name, project start and completion dates, total job hours, auditor name)
- Energy usage and improvement information (e.g., original energy consumption [use and dollars], energy savings estimate [energy units and dollars], name of electricity and gas provider).
When making design decisions, the most common models, sometimes combined, are those in which the program:
- Sponsors an existing loan product or program
- Provides capital, either to lend directly or as a credit enhancement.
Under the sponsorship model, your program partners with one or more lenders that originate and service loans. These institutions manage their lending activities with loan origination systems and servicing systems. As a sponsor of their loan product, you can request weekly or monthly reports that include information on applications, approvals, originations, and servicing activity.
Under the model in which you provide capital to fund loans, you will have a close relationship with your lending partners and can request access to their origination and servicing systems to access data such as applications received, applications approved and denied, etc.; after all, it is your capital at risk.
Under the model in which your program provides a credit enhancement, you can request weekly or monthly reports from the lender, similar to the sponsorship model. Regardless of the type of credit enhancement you are providing (e.g., loan loss reserve, interest rate buy-down, debt service reserve, subordinate capital), you will need detailed reporting to track the use of your funds. Each of these credit enhancements will use funds at different times during a transaction and for different purposes; consequently there are numerous options for managing this data:
- A loan loss reserve is possibly the most complex and requires a management process and ideally an automated system through which lenders can place requests for a loss reserve amount and receive a confirmation of that commitment to fund the loan loss reserve. This process could also be used to allow the financial institution(s) to process claims against the fund and to recover losses from defaulted loans. Preferably, your program can partner with an independent financial institution to manage this function, but if not it will need to become part of your program’s accounting activities.
- An interest rate buy-down is a one-time per loan payment to the financial institution. Data from this transaction can be tracked without elaborate systems, and your lender can provide a full data report.
- Debt service coverage payments are paid to the owner/investor of the loan when delinquencies interrupt the flow of payments and these funds are returned to the debt service account when payments are made or the loan is charged off. As such, tracking this activity can only be performed by the loan servicer’s system. Consequently, you need comprehensive reporting from your servicer to insure that you can properly track the use of your funds.
- Subordinate capital is essentially an investment in another party’s loan portfolio, and you will receive investor reporting from your lender.
In addition to accessing financial data on borrowers and the origination and servicing of loans, you may choose to collect property, project, and energy performance data. Contractors can provide property and project data that can typically be delivered to the lender as part of the loan origination process. Alternatively, contractors can send this data directly to your program to avoid over burdening the lender, as some lenders prefer to only provide data related to the loan itself. It may be possible to also collect energy data (i.e., energy consumption before and after the upgrade) but doing so will require a consent waiver signed by the borrower and the approval and support from the utility company. The combination of financial and energy project data will allow you to evaluate cost effectiveness and explore possible correlations among borrower, property, project, energy performance and financial performance.
Keep in mind that certain data (e.g., name, address, social security number) are considered personally identifiable information (PII), and the handling of this data is highly regulated. If for any reason you need to share the PII data that you collect, you will need to acquire a legally viable consent waiver from each borrower. Consequently, you will need to review and understand the state and federal regulations related to data collection and sharing. State regulations can be identified by contacting your state’s attorney general and the banking division. Federal regulations are contained in Gramm, Leach, Bliley legislation. You may need external expert advice to ensure your program complies with these regulations.
Your approach to transferring (i.e., between capital providers, lenders, investors, and your program), storing, and managing data will depend on the size of your program, your information systems budget, and your evaluation needs. If you intend to capture less than 20 data elements for fewer than one or two thousand loans, a conventional spreadsheet may be sufficient to capture and transfer data. The spreadsheet approach has a very low initial cost, but maintaining it is a manual process and will require substantial, regular staff time that will increase as program volume grows.
If you have greater resources, and data needs, your program will need to include an information technology staff person (or subcontractor) who will help you identify data management options for building and maintaining a database. There are generally two key steps in the developmental effort.
- The first step is to create the data schema, a listing of the data elements, their definitions and the file format for transferring and storing the data.
- The second step is to develop the data platform that allows financial institutions or other data providers and users to link to your database using automated programming interfaces (API) and servers to store, manage and access the data.
In most cases, developing a database with functionality beyond the use of simple spreadsheets will exceed the resources of most programs. The U.S. Department of Energy has developed the Building Energy Data Exchange Specification (BEDES) and the Standard Energy Efficiency Data platform (SEED). These two DOE systems will allow you to adopt a pre-developed data schema and data platform, respectively.
See the Evaluation & Data Collection’s Develop Resources handbook for information on how to transfer, store, and manage your program’s data.
Effective Data Collection for Energy Efficiency Lending
More than 200 energy efficiency loan programs in 49 states are administered by utilities, state/local government agencies, or private lenders. This distributed model for efficiency financing has led to significant variation in program design and implementation practices including how data is collected and used.
Energy Efficiency Finance Programs: Use-case Analysis to Define Data Needs and Guidelines, issued by the State and Local Energy Efficiency Action Network's Financing Solutions Working Group and prepared by Lawrence Berkeley National Laboratory, makes the case for establishment of common data collection practices for energy efficiency lending. The report can be used by two groups:
- Program administrators and policy makers who can learn lessons about more effective program design from the comparative analysis of data from diverse program types and approaches.
- Lenders and investors who can use loan performance data to more accurately account for risk and thereby potentially increase lending and lower financing costs. Consistent data collection may also provide an enhanced ability to sell loan pools to replenish program funds.
The report covers:
- Rationales for collecting more consistent data from energy efficiency finance programs
- Discussion of energy efficiency finance program use cases
- Challenges with collecting information from customers who participate in finance programs
- Issues with data consolidation and aggregation across multiple finance programs.
In addition to identifying the data you intend to collect, your evaluation plan should also establish the process and schedule for reviewing and sharing data with your program partners and stakeholders. An effective approach is to develop a standard reporting format and produce a monthly report, which can be shared with your partners, including lenders, capital providers, contractor managers, inspection services, and other stakeholders. After providing reports, you can schedule reviews and assessment sessions to identify and communicate findings from the reporting and to consider program modifications.
More information about the review and assessment of your financing activities is available in the Financing-Assess & Improve Processes handbook.
Tips for Success
In recent years, hundreds of communities have been working to promote home energy upgrades through programs such as the Better Buildings Neighborhood Program, Home Performance with ENERGY STAR, utility-sponsored programs, and others. The following tips present the top lessons these programs want to share related to this handbook. This list is not exhaustive.
Measuring performance at key points in the upgrade process (e.g., assessments, conversion rates, and financing applications) has helped programs understand where their processes are working smoothly and where they are not. This information has helped them continuously improve their program design and implementation. To monitor progress, successful programs have combined information from their project tracking systems with customer surveys, information from call centers, and feedback from contractors and lenders to understand the customer experience. Make data accessible for program staff to track progress, identify successful strategies, and detect points of failure.
- Enhabit, formerly Clean Energy Works Oregon, established an extensive process for getting customer feedback at key points in the program delivery process to evaluate customer satisfaction and better understand why some homeowners chose to undertake upgrades while others did not. The program identified seven points in the program delivery process to gather information through feedback surveys and phone interviews: application, assessment, bid, drop-out, financing, completion, and experience after 12 months. The program credited this kind of customer communication and feedback as one of the keys to its ongoing success.
Source: Clean Energy Works Research Planning, Will Villota, CEWO, 2012 (Presented during January 19, 2012 Better Buildings Residential Neighborhood Program peer exchange call).
- Boulder County’s EnergySmart program sent an online customer feedback survey to homeowners who had completed upgrades. Among other things, the customer surveys affirmed customer satisfaction and identified the opportunity for word-of-mouth marketing. Surveys found that the vast majority of the respondents would recommend the EnergySmart service to a friend or neighbor. The surveys also surfaced some weaknesses that the program resolved. For example, some respondents noted contractor’s lack of response and professionalism as an issue, which led the program to develop guidelines for professionalism and customer contact. Surveys also noted that the assessment report was long and confusing, leading the program to develop a new, customized report that was easier to follow and clearer about next steps.
- Connecticut’s Neighbor to Neighbor Energy Challenge used qualitative contractor and customer feedback combined with quantitative data to evaluate how well its outreach efforts led to home energy assessments. When informal contractor feedback alerted program managers that relatively few interested customers were following through to have assessments conducted on their homes, the program analyzed project data and found that only around a quarter of customers who expressed interest in an assessment had completed one. To diagnose the problem, the program analyzed data to see how customers were acquired, how long it took to send leads to contractors, and how long it took contractors to follow up with customers to arrange for an assessment. Through qualitative analysis, the program found, among other things, that customers didn’t understand what they were signing up for and may have been unwilling to say “no” to young and enthusiastic outreach staff. The program also found that its staff wasn’t following up quickly enough with people that wanted more information. In response, the program improved its process for distributing leads to contractors (e.g., linking contractors to homeowners in 1-2 days), created a “receipt” for interested customers outlining next steps, and set up a system to call non-responsive leads after two weeks. With these and other steps, the program increased its close rate 35% in one month after changes were implemented.
The following resources provide topical information related to this handbook. The resources include a variety of information ranging from case studies and examples to presentations and webcasts. The U.S. Department of Energy does not endorse these materials.
This report details findings from the evaluation of the Colorado energy efficiency financing program.
Evaluation of the Efficiency Maine Trust Residential Direct Install Program: Final Evaluation Report
This report proposes a set of data elements that should be collected for each residential and non-residential project, including borrower, property, project, and financing information at the time of the installation. The data set also includes post-installation information on the performance of both the financing and the project. These data elements were selected based on an assessment of various data “users” (financial institutions, policy makers, vendors, program administrators, etc.) and their “uses” for the data (what questions would the users want the data to answer). The report also sets out the sources of the data and a methodology for collecting, managing and providing access to the data, and addresses privacy issues and the use of data anonymization and aggregation.
Key Findings and Recommendations From the Process Evaluation of Clean Energy Works Oregon (now Enhabit)
This report presents key findings and recommendations from the process evaluation of Clean Energy Works Oregon's (now Enhabit's) energy efficiency financing program. Table 1 provides a good list of key process evaluation research questions which may help others scope comprehensive process evaluations.
This summary from a Better Buildings Residential Network peer exchange call focused on how loan performance data is tracked and analyzed, and what the data shows.
Presentation providing an overview of financing programs, a strategy for continuous improvement, tools for program management, a risk management strategy, and common risks associated with financing programs.
This worksheet can help you organize your ideas and methods for creating an effective evaluation plan.
Making it Count: Understanding the Value of Energy Efficiency Financing Programs Funded by Utility Customers
This report lays the groundwork for a dialogue to explore regulatory and policy mechanisms for ensuring that efficiency financing initiatives provide value for society and protection for consumers. Through case studies of Connecticut, New York, Massachusetts, California, and Maryland, it explores emerging issues that jurisdictions will need to tackle when considering an increased reliance on financing.
This paper examines criteria for a comparative assessment of multiple financing programs for energy efficiency, developed through a statewide public process in California.
This article discusses the importance and value of evaluating energy efficiency financing programs.
This website provides an overview of financing as it pertains to state, local, and tribal governments who are designing and implementing clean energy financing programs. Residential financing tools include residential PACE (R-PACE), on-bill financing and repayment, loan loss reserves and other credit enhancements, revolving loan funds, and energy efficient mortgages.
This U.S. Environmental Protection Agency resource is intended to help state and local governments design finance programs for their jurisdiction. It describes financing program options, key components of these programs, and factors to consider as they make decisions about getting started or updating their programs.