In this handbook, you will learn how to establish a plan to regularly evaluate your market position and revisit the initial market assessment that you conducted. This is important for two reasons:
- You need to understand how key stakeholders view your organization. Knowing whether you are delivering services that meet their expectations and whether they will continue to support or engage with your organization is important to its long-term viability.
- You need to understand whether you have new competitors that may require a change in your overall approach or business plan in order to maintain your organization’s relevance in the market.
As part of developing your evaluation plan, you will learn how to:
- Reevaluate your organization’s mission, vision, and goals, as needed to ensure they still align with the role your organization could play in the current market
- Revisit your business model and explore how each area is functioning
- Stay aware of policy changes which can influence the market and the role your organization plays in it
- Actively manage risks and leverage opportunities that may allow your organization to grow and broaden its reach.
Step by Step
To understand and monitor progress and success, you need to develop a plan for taking a step back from day-to-day business operations to revisit your market assessment and evaluate your current operations. You should reassess:
- Potential customers
- Funding sources.
Based on the information you gain from your evaluation, you should consider whether any changes are needed in your organization’s:
This should be done at least annually, and potentially more often as you establish your organization’s position in the market and strengthen your organization’s ability to meet its goals.
The six steps below provide details on the pieces that should be included in your plan for scanning your external environment and regularly evaluating your market position within that environment.
The first piece of your evaluation plan should be reassessing your market position on a regular basis to ensure you are delivering services that are both needed and wanted. You also need to understand whether you have new competing resources that may require a change in your overall approach or business plan in order to maintain your organization’s relevance in the market and long-term viability.
In developing your initial market assessment, you spent time on:
- Assessing the nature of market demand for energy efficiency products and services
- Assessing how the market is already being served by other organizations and what gaps exist
- Assessing your organization’s strengths, capabilities, potential partners, and constraints in providing needed products and services in the market.
When you develop your evaluation plan, you should first reconsider your market assessment, focusing in particular on identifying any changes in the competitive landscape. Think about each area of your original assessment:
- Potential market demand for energy efficiency products and services
- What new information have you received or market changes have occurred (e.g., has market demand for energy efficiency upgrades drastically increased or decreased)?
- How the market is being served by other organizations
- Are other organizations seeking to serve the same stakeholders you are, either with energy efficiency services or other services that could compete for your stakeholders’ time and attention?
- Have any previous competitors dropped out of the market or shifted focus?
- Have any new potential partners entered the market?
- Your strengths and capabilities to provide products and services in the market
- Do you have new staff with expanded expertise in any areas?
- Has your range of service offerings changed in any way?
After gathering information on your market position, the second piece of your evaluation plan should be evaluating your organization’s vision, mission, and goals to ensure they still align with the role your organization could play in the current market. It is unlikely that your vision and mission statements will need annual updates—unless there has been a very significant change in policy, the home performance market, or your key competitors—but your goals may need to be updated.
For example, a reasonable goal for an organization administering a home energy upgrade program is to create 50 new jobs in a community in 3 years. If you find that your organization has already achieved 75% of the goal in the first year, consider increasing your goal or adding another goal to provide a clear focus for your organization’s efforts. Similarly, if you are far below being on track to meet your original goal, you will want to consider revising your goal to something more realistic (e.g., 30 new jobs by year 3 or 50 new jobs by year 6).
When reassessing your progress and goals, consider how external factors may have affected your successes or challenges. For example, if you are not on track to meet your annual homeowner sign-up goal,
- Has a new energy efficiency program entered your market, and do your programs overlap too much?
- Has a lead generating partner organization had staff turnover that reduced its level of activities, resulting in fewer homeowner referrals than anticipated?
If your organization exceeds its homeowner sign-up goal,
- Could a portion of them be attributed to a local policy change?
- Did your program received high-profile press coverage?
Ask yourself whether external factors such as these impacted progress towards your program goals, and consider whether or not they will be in place in future years, before adjusting similar future goals.
The next step in your annual evaluation plan should be to revisit your business model to assess how efficient your operations are and whether your strategies and tactics are leading you to meet your organizational goals and effectively run your residential energy efficiency program.
When you developed your business model, you determined the following:
- What services to provide and how you will provide them
- The customers to whom you will provide services
- The assets and infrastructure you will need to implement your program
- Your financial model
- Your governance structure.
As part of your evaluation, you should explore how well each of these areas is currently functioning. This is a good way to understand whether any assumptions you made need to be updated and whether any small adjustments in your internal processes or larger modifications to your business model are necessary. See the Market Position handbook on Assessing & Improving Processes for more details on how and what metrics to consider.
Evaluating Your Organization’s Governance Structure
An organization’s governance structure is a critical component of its success that may need to be updated over time, particularly if there are changes in your organization’s partners, stakeholders, or leadership team.
For example, if your organization is a nonprofit with an executive director reporting to a board of directors, your business model may have been based on the assumption that the board would meet quarterly to conduct its business. In your review of the past year’s operations, however, you may find that the board met more often via email and ad hoc conference calls. This information may indicate a need for more frequent standing meetings, or for subcommittees (e.g., executive committee, audit committee) to be established in order for the organization’s work to continue at an appropriate pace.
Next, your evaluation plan needs to explore policy changes that can influence the market, and the role your organization plays. Staying aware of the policy environment will allow your organization to react to any significant changes such as:
- Municipal or utility energy efficiency rulemakings
- Point of sale disclosure regulations
- Property assessed clean energy (PACE) enabling legislation
- Green jobs initiatives.
Any one of these, or related, changes may trigger a need for a shift in organizational emphasis or the services you offer.
For example, legislation enabling PACE financing could increase homeowners’ ability to pay for upgrades. As a result, your organization might want to create a new initiative or program designed to increase homeowner outreach about new financing options for energy efficiency.
Vermont Legislation Allows for Increased Service
In 2009, the state of Vermont enacted legislation that enabled towns to establish PACE districts. Creating these districts was the first step in making PACE financing available to homeowners who can access the funds to make home energy improvements, which are then paid back through property tax bills.
As of early 2016, 53 towns in the state had set up PACE districts. The statewide residential energy efficiency program administrator, Efficiency Vermont, leveraged this activity by allocating staffing and developing resources - such as a Wiki site that included sample documents, tools, and a discussion board - to encourage towns to take the next step of adopting the PACE program and to support the towns during program implementation.
By monitoring changes in its external operating environment, Efficiency Vermont was able to add new consulting services such as acting as a PACE administrator for towns, partnering with credit facilities to accept PACE applications from customers, and improving customer outreach and communications around the availability of PACE financing.
Risk management is defined as the process of forecasting and evaluating risks to an organization and identifying the steps needed to avoid or minimize their impact. You evaluation plan should include a strategy for working closely with your network of business partners and stakeholders to carefully identify risks to your organization’s ability to meet its goals and objectives. These risks can include:
- A change in political environment
- Policy changes that affect your business
- Negative media coverage of your organization
- A new competitor for the attention of your target customer audience.
Your plan should also include a strategy for developing ideas to either mitigate or address any risks you identify.
In addition to determining how you will manage risks, your evaluation plan should also explain how you will identify and leverage opportunities that may allow your organization to grow and broaden its reach.
For example, based on the updated market assessment that you conducted, you may find that a partner organization has received new grant funding to support workforce development activities. That information could lead to an opportunity for your organization to access some of those funds and broaden your own contractor training activities. Or, you may discover a staffing need that could be filled with a new hire that brings contacts and relationships upon which your organization can build.
As you put together the pieces of your plan, think through what amount of information will be needed to enable you to make insightful decisions. Note that there often is a tradeoff between more and better information and the cost of acquiring that information. The key is to determine the level of best available information that will allow you to make well-informed decisions.
Before making any organizational changes:
- Identify what information you will need to make effective decisions
- Evaluate if you have enough data to inform a decision
- If you do not, consider what additional inputs you need and where you will get them
In the Market Position handbook on assessing and improving processes, you will learn how use the information from your evaluation plan to determine what changes, if any, are needed to enhance the effectiveness of your organization.
Tips for Success
In recent years, hundreds of communities have been working to promote home energy upgrades through programs such as the Better Buildings Neighborhood Program, Home Performance with ENERGY STAR, utility-sponsored programs, and others. The following tips present the top lessons these programs want to share related to this handbook. This list is not exhaustive.
To develop a successful business model, Better Buildings Neighborhood Program partners found it critical to have a strong understanding of the external environment within which they operated. This included who their customers were, who their competitors and partners were, what key policies governed their work, and what trends were likely to impact their ability to accomplish their goals and fulfill their mission. Understanding the external forces that affected their market allowed the organizations to better identify services that met customers’ and partners’ needs and develop a more robust business model. Many organizations adapted their business model to overcome challenges and leverage opportunities as local conditions and their understanding of how their business operated within the market evolved.
- When the New Hampshire Better Buildings program began, there were numerous energy efficiency programs operating independently in the state (e.g., by multiple utilities), and only informal coordination among activities was occurring. Customers were often overwhelmed trying to determine which programs they qualified for, how to fill out all the required paperwork, and which contractors they were able to work with. The program quickly learned that their business model needed to be well-integrated with other energy efficiency programs in the state, in order to provide value to customers and contractors while enabling long term growth for the efficiency market.
In response, the program took a collaborative approach to their business model design and partnered with the utilities’ rate-payer funded Home Performance with ENERGY STAR (HPwES) program. Combining programs leveraged the utility programs’ existing queue of upgrade projects, procedure for assessments and upgrades, and database for collecting information about each project. A single program and process also made the most sense for residential customers, who were more likely to move forward with a project if they did not have multiple programs and processes to figure out. The utility companies benefitted from the partnership because the additional funds from Better Buildings allowed them to expand their program’s reach. Combining efforts and utilizing each program’s strengths led to consistent marketing and messaging, more efficient processes for contractors, one-stop shopping for customers, and a streamlined approach to financing.
- The Neighbor to Neighbor Energy Challenge’s (N2N) original program design used community-based social marketing (CBSM) to acquire and feed leads into the existing ratepayer funded Home Energy Solutions (HES) assessment program. N2N expected that contractors would convince customers to take advantage of rebate programs to complete home energy upgrades; however, over the first two years of the program, they saw many of the leads they generated stall after the assessments. N2N realized that they had limited influence over the contractor network with this approach and that the HES program design did not incent contractors and customers to complete upgrades. They shifted their business model and focused marketing and outreach resources on new strategies, such as direct lead acquisition, in order to acquire customers who were more likely to proceed straight to completing upgrades.
The following resources provide topical information related to this handbook. The resources include a variety of information ranging from case studies and examples to presentations and webcasts. The U.S. Department of Energy does not endorse these materials.
This report for the Connecticut Energy Efficiency Board provides a review of best practices in impact evaluation, recommendations for calculating oil and propane savings, and discusses the impact evaluation findings for the Home Energy Services (HES) and Home Energy Services-Income Eligible (HES-IE) Programs. This best practices review provides an overview of key evaluation protocol and guideline documents.
This presentation includes a brief Introduction to Northeast Energy Efficiency Partnerships (NEEP) and the Regional Evaluation, Measurement and Verification Forum (EM&V Forum), Regional Energy Efficiency Database (REED) development and content, and how to access the REED reports and underlying data.
This webinar series is intended for state officials starting or expanding their EM&V methods for a wide range of efficiency activities including utility customer-funded programs, building energy codes, appliance and equipment standards, energy savings performance contracting, and efficiency programs that support pollution reduction goals or regulations.
Presentation describing how Conservation Services Group uses data to monitor market transformation and for internal QA/QC purposes.
This worksheet can help you organize your ideas and methods for creating an effective evaluation plan.
The U.S. Department of Energy's Better Buildings Residential Program released version 2.0 of a user-friendly tool for estimating the cost-effectiveness of a residential energy efficiency program based on program administrator inputs. Cost-effectiveness analysis compares the benefits (i.e., outputs or outcomes) associated with a program or a measure with the costs (i.e., resources expended) to produce them. Program cost-effectiveness is commonly used by public utility commissions to make decisions about funding programs or program approaches. Program designers, policy makers, utilities, architects, and engineers can use this tool to estimate the impact of different program changes on the cost-effectiveness of a program.
REED serves as a dashboard for the consistent reporting of electric and natural gas energy efficiency program energy and demand savings and associated costs, avoided emissions and job impacts across the Northeast and Mid-Atlantic region. REED is a project of NEEP's Regional Evaluation, Measurement and Verification Forum (EM&V Forum) and is based on the EM&V Forum's Common Statewide Energy Efficiency Reporting Guidelines.
Tool to evaluate contractor impacts on program revenue.
The State and Local Energy Efficiency Action Network (SEE Action) Evaluation, Measurement, and Verification (EM&V) Resource Portal serves as an EM&V resource one-stop shop for energy efficiency program administrators and project managers. The resources focus on tools and approaches that can be applied nationwide, address EM&V consistency, and are recognized by the industry.
The Residential Retrofit Program Design Guide focuses on the key elements and design characteristics of building and maintaining a successful residential energy upgrade program. The material is presented as a guide for program design and planning from start to finish, laid out in chronological order of program development.