U.S. Department of Energy Energy Efficiency & Renewable Energy U.S. Department of Energy Energy Efficiency & Renewable Energy

Description

Key Resources

None available at this time.

While evaluation can seem like a final step in a program’s marketing and outreach efforts, designing for evaluation from the start enables you to adjust marketing midstream as needed, focus efforts on your more successful strategies, and enhance early outreach successes. An evaluation plan helps measure progress toward achieving program goals and marketing and outreach objectives. It also helps you determine how the return on investment in one marketing strategy compares to your other available strategies.

Building evaluation planning into your marketing and outreach efforts means identifying key data points to track before launching your marketing and outreach. It also means collecting marketing data in conjunction with overall program data collection and evaluation. See the Program Design Develop Evaluation Plans handbook for more detail about data collection and key marketing and outreach metrics to establish and track as part of overall program evaluation efforts.

Consider how to evaluate the effectiveness of your program’s different marketing and outreach approaches in a variety of ways, whether it is the number of leads, assessments, and upgrades gained from a particular strategy or the cost-effectiveness of a particular initiative. This information will help you make adjustments as needed and support continuous improvement in marketing and outreach.

Evaluating and adapting marketing and outreach efforts throughout the process allows programs to achieve their goals and objectives by applying lessons learned to continuously improve and avoiding strategies that have not demonstrated results nor been cost-effective.

The steps to developing an effective marketing and outreach evaluation plan include:

  • Identify metrics for each marketing and outreach objective and how each will be evaluated
  • Draft a marketing and outreach evaluation plan
  • Share your evaluation plan with partners.
Find related information across other program components:

Step-by-Step

To evaluate your program’s marketing and outreach efforts, you need to first identify the key metrics to measure progress toward achieving goals and objectives, then draft a marketing and outreach evaluation plan to analyze those metrics.

The Driving Demand for Home Energy Improvements report notes that designing for evaluation and data collection from the start enables you to adjust your program midstream as needed, focus efforts on the most effective marketing strategies, and enhance early program successes. Evaluation approaches will vary based on your organization’s marketing activities, metrics, and reporting requirements.

These steps will help your program develop a plan for evaluating your marketing and outreach:

Identify metrics and evaluation criteria for each marketing and outreach objective

Before planning to evaluate your progress, determine which marketing and outreach metrics to track. Metrics are measurable indicators of the success of your efforts in reaching your marketing and outreach goals and objectives. They should be specific and quantitative.

Your program should select indicators to best evaluate how you are meeting marketing and outreach objectives. Following are some of the possible metrics to measure success of various marketing and outreach approaches:

  • Number of leads, assessments, and upgrades associated with specific marketing campaigns
  • Cost per lead achieved through a specific outreach channel
  • Conversions of leads to assessments and upgrades
  • Effectiveness of sales training or contractor marketing based on contractor success rates
  • Percentage of customers taking advantage of incentives based on an outreach campaign
  • Other measures of engagement, including:
    • Advertising response rate (e.g., as tracked by a promotional code included in advertisements)
    • Responses to a particular call to action (e.g., call an energy advisor)
    • Success rate of door-to-door canvassing (e.g., as reported by canvassers or sign-ups)
    • Number of leads from an outreach event that resulted in assessments or upgrades

Promotional Codes Provide Ways to Track Leads

Measuring the impact of a specific marketing tactic (e.g., paid radio advertising or a direct mail campaign) becomes more complicated when your marketing and outreach efforts are many and varied. Adding promotional codes unique to an event or information source to marketing materials and to request the code when customers follow the call to action is one useful tactic for tracking how customers are brought to the program.

Enhabit, formerly Clean Energy Works Oregon, used coupon codes in its advertising materials to help its contractor partners identify where the homeowner learned about the program, and to track the efficacy of various outreach materials. Promotional codes on materials advertising a special “buy-down” financing incentive, where customers had the option of financing a portion of the upgrade cost with low monthly payments, also gave a sense of urgency and “specialness” to the offer.

Promotional Flier Created by Enhabit Contractor

Abacus Energy Solutions, LLC, an Enhabit contractor, developed this flier that included a promotional code to be entered on Enhabit’s website.

Abacus Energy Solutions, LLC flier

Source: Enhabit, 2011.

When selecting your marketing metrics:

  • Determine a baseline or starting point for each marketing metric. Your metrics should be measurable over time in order to monitor progress
  • Take into account factors that might affect your results, such as partners’ simultaneous marketing efforts
  • Remember that marketing tactics might not always result in actual leads, but they can help increase program exposure, so try to count that metric as well.

When identifying indicators of marketing effectiveness, you can look at all marketing-related costs to determine a total cost of your marketing activities per upgraded home. Strategies may be creative or win accolades, but they are only effective if they make good use of resources by building interest in or bringing leads to your program. Consider evaluating all marketing campaign costs, which can include the following:

  • Direct and indirect staff time for any work related to marketing and outreach activities (e.g., strategic planning, writing, editing, graphics, helplines, event staffing)
  • Consultants (e.g., public relations, marketing, website design, creative direction)
  • Financial incentives (e.g., rebates, discounts, one-time offers)
  • Development of marketing materials (e.g., branding, advertising, printed items, give-aways)
  • Other marketing-related expenses (e.g., direct mail postage, advertising placement, events, website maintenance)

This approach helps determine how the return on investment in one marketing strategy compares to your other available strategies. One of the biggest barriers that programs encounter when evaluating outreach efforts is realizing that there is no one solution when it comes to marketing, and that it may take several different approaches to achieve the best results from your marketing investment.

Evaluation Plan Helps Program Leap to Success

Data tracking helped the Local Energy Alliance Program (LEAP) in Charlottesville, Virginia, learn that one direct mailing campaign delivered 405 leads to the program. These leads resulted in 52 home energy upgrades. By tracking the cost of the campaign, leads, and completed upgrades related to the direct mailings, LEAP was able to determine how much it spent per upgrade brought into the program using direct mailings. The program collected this type of information for all of its marketing efforts. In this way, LEAP could assess whether or not the strategy was worth its return on investment by comparing the leads generated from various marketing strategies to the number of upgrades that occurred as a result of each strategy.

Close

Draft a marketing and outreach evaluation plan

The evaluation plan you develop should align with marketing and outreach objectives, as well as your program’s overall evaluation plan and data collection efforts. It should include a list of measurable indicators and outline who, when, and how you will measure marketing and outreach results:

  • Who: As part of your plan, define who will collect the specific marketing data and how they will use it to measure results. Make sure to include roles and responsibilities for your program staff and partners in your plan. Coordinate your marketing and outreach data collection with any overall program data collection efforts, so that different people from your program are not asking the same sources for the same information at different times.
    • A local university might be an effective data collection partner because undergraduate or graduate research students can be enlisted to conduct awareness surveys, compile data, and track results.
  • When: Depending on the metric, you may be able to analyze results immediately (e.g., number of homeowners that sign up for energy assessments at an event) or not until several months later (e.g., increase in website visitors due to online advertisements). Be patient, but plan to review and analyze your data as it becomes available. The timing of results from your marketing and outreach evaluation will depend on what metrics you are tracking. If your evaluation plan covers more long-term activities, make sure you are not missing an opportunity to track interim milestones, such as mid-campaign results or quarterly analyses. Your marketing evaluation plan will also feed into the overall program data collection and evaluation plan, so make sure to coordinate your efforts with relevant program staff.
  • How: Your plan should establish the processes needed to collect data related to your metrics, including promotional material codes, survey mechanisms, media monitoring services, social media trackers, website analytics, and customer databases. Certain data collection could also require up-front agreements or customer signatures.
    • For example, Philadelphia EnergyWorks in its data-sharing partnership with Philadelphia Gas Works (PGW), made sure the proper requirements were observed for collecting customer data and that customers signed their data sharing agreement forms. By sharing effectiveness data from its specific marketing efforts, EnergyWorks helped to sustain PGW’s residential energy efficiency efforts into the future by showing what tactics worked most effectively to encourage program participation.
  • The Better Buildings Neighborhood Program’s Creating an Evaluation Plan Worksheet can help organize your thoughts and methods for creating an effective evaluation plan for your marketing and outreach efforts. The sample matrix below has been filled out with examples, and it shows how the evaluation plan tracks with marketing and outreach-related goals and objectives.

Sample Marketing & Outreach Evaluation Plan Matrix

Source: U.S. Department of Energy, 2014.

Close

Share your evaluation plan with partners

Once you have developed a marketing and outreach evaluation plan, you can engage relevant partners to determine if they have any suggestions for efficiently collecting marketing and outreach data, or are able to support the program. Tapping into their knowledge and resources can help save time and effort in the long run.

Close

Tips for Success

In recent years, hundreds of communities have been working to promote home energy upgrades through programs such as the Better Buildings Neighborhood Program, Home Performance with ENERGY STAR, utility-sponsored programs, and others. The following tips present the top lessons these programs want to share related to this handbook. This list is not exhaustive.

Measure and evaluate performance at key points in the process

Measuring performance at key points in the upgrade process (e.g., assessments, conversion rates, and financing applications) has helped programs understand where their processes are working smoothly and where they are not. This information has helped them continuously improve their program design and implementation. To monitor progress, successful programs have combined information from their project tracking systems with customer surveys, information from call centers, and feedback from contractors and lenders to understand the customer experience. Make data accessible for program staff to track progress, identify successful strategies, and detect points of failure.

  • Enhabit, formerly Clean Energy Works Oregon, established an extensive process for getting customer feedback at key points in the program delivery process to evaluate customer satisfaction and better understand why some homeowners chose to undertake upgrades while others did not. The program identified seven points in the program delivery process to gather information through feedback surveys and phone interviews: application, assessment, bid, drop-out, financing, completion, and experience after 12 months. The program credited this kind of customer communication and feedback as one of the keys to its ongoing success.

CEWO Feedback Surveys

Source: Clean Energy Works Research Planning, Will Villota, CEWO, 2012 (Presented during January 19, 2012 Better Buildings Residential Neighborhood Program peer exchange call).

  • Boulder County’s EnergySmart program sent an online customer feedback survey to homeowners who had completed upgrades. Among other things, the customer surveys affirmed customer satisfaction and identified the opportunity for word-of-mouth marketing. Surveys found that the vast majority of the respondents would recommend the EnergySmart service to a friend or neighbor. The surveys also surfaced some weaknesses that the program resolved. For example, some respondents noted contractor’s lack of response and professionalism as an issue, which led the program to develop guidelines for professionalism and customer contact. Surveys also noted that the assessment report was long and confusing, leading the program to develop a new, customized report that was easier to follow and clearer about next steps.
  • Connecticut’s Neighbor to Neighbor Energy Challenge used qualitative contractor and customer feedback combined with quantitative data to evaluate how well its outreach efforts led to home energy assessments. When informal contractor feedback alerted program managers that relatively few interested customers were following through to have assessments conducted on their homes, the program analyzed project data and found that only around a quarter of customers who expressed interest in an assessment had completed one. To diagnose the problem, the program analyzed data to see how customers were acquired, how long it took to send leads to contractors, and how long it took contractors to follow up with customers to arrange for an assessment. Through qualitative analysis, the program found, among other things, that customers didn’t understand what they were signing up for and may have been unwilling to say “no” to young and enthusiastic outreach staff. The program also found that its staff wasn’t following up quickly enough with people that wanted more information. In response, the program improved its process for distributing leads to contractors (e.g., linking contractors to homeowners in 1-2 days), created a “receipt” for interested customers outlining next steps, and set up a system to call non-responsive leads after two weeks. With these and other steps, the program increased its close rate 35% in one month after changes were implemented.
Close

Examples

The following resources are examples from individual residential energy efficiency programs, which include case studies, program presentations and reports, and program materials. The U.S. Department of Energy does not endorse these materials.

Case Studies

  1. Author: U.S. Department of Energy
    Publication Date: 2012

    This case study discusses BetterBuildings for Michigan's targeted outreach campaigns which applied varying incentives and outreach strategies to neighborhoods with a goal to understand which rebates and strategies work best in the target communities.

  2. Author: U.S. Department of Energy
    Publication Date: 2012

    This case study describes Efficiency Maine's Home Energy Savings Program (HESP), one of the few large residential energy efficiency programs that has attempted to navigate the transition from rebate-focused offerings to financing focused offerings that better align with its limited budget.

  3. Author: U.S. Department of Energy
    Publication Date: 2013

    This case study discusses Energy Impact Illinois' marketing evolution from a broad outreach campaign to a Òhouse partyÓ approach that brought Chicago homeowners, neighbors, and friends together to learn about energy efficiency opportunities, while increasing demand for home energy assessments and upgrades.

  4. Author: U.S. Department of Energy
    Publication Date: 2013

    Philadelphia EnergyWorks helped sustain future programs by sharing marketing insights and program data with a local utility partner.

Program Presentations & Reports

  1. Author: Opinion Dynamics Corporation
    Publication Date: 2016

    The purpose of this study is to document the effects of California’s Statewide Marketing, Education and Outreach (SW ME&O) program. The primary objective of this study is to assess the effectiveness of SW ME&O efforts overall, as well as against established performance metrics.

  2. Author: Opinion Dynamics Corporation
    Publication Date: 2016

    The presentation covers evaluation results and recommendations for California’s Statewide Marketing, Education and Outreach (SW ME&O) program.

  3. Author: The Cadmus Group, Inc.
    Publication Date: 2012

    NeighborWorks of Western Vermont (NWWVT) contracted with The Cadmus Group, Inc., to evaluate its H.E.A.T. Squad program. The evaluation activities informed two main areas of interest: program and market effects, and impact and cost-effectiveness. To inform the evaluation, Cadmus surveyed participant and non-participant homeowners and interviewed program stakeholders.

  4. Author: Local Energy Alliance Program
    Publication Date: 2012

    This presentation explains the pro forma spreadsheet used by Virginia's Local Energy Alliance Program to evaluate program impact.

  5. Author: Marlowe Kulley, Portland Bureau of Planning & Sustainability
    Publication Date: 2011

    This presentation is a tour of the project evaluation and data collection system that Clean Energy Works Portland uses to survey its participating residents.

  6. Author: Kerry O'Neill, Connecticut Neighbor to Neighbor Energy Challenge
    Publication Date: 2011

    This presentation shares how the Neighbor to Neighbor Energy Challenge collected and evaluated data and used the results to improve its program.

  7. Author: Dimitrios Laloudakis, Energize Phoenix
    Publication Date: 2011

    This presentation outlines the techniques for collecting and evaluating energy efficiency program evaluation data, including data related to marketing efforts.

Program Materials

  1. Author: Consumer Federation of America
    Publication Date: 2016

    This marketing press release describes 12 tips for consumers to save energy and money in celebration of the first Energy Efficiency Day.

Toolbox

The following resources are available to help design, implement, and evaluate possible activities related to this handbook. These resources include templates and forms, as well as tools and calculators. The U.S. Department of Energy does not endorse these materials.

Templates & Forms

  1. Author: U.S. Department of Energy
    Publication Date: 2014

    This worksheet can help you organize your ideas and methods for creating an effective evaluation plan.

  2. Author: U.S. Department of Energy
    Publication Date: 2011

    This sample email survey template, created by the Better Buildings Neighborhood Program, was designed for programs to develop their own survey of successful program participants in order to assess customer experience.

  3. Author: U.S. Department of Energy
    Publication Date: 2011

    This sample phone survey template, created by the Better Buildings Neighborhood Program, was designed for programs to use with applicants who have been screened out from participating in a program.

Tools & Calculators

None available at this time.

Topical Resources

Topical Presentations

  1. Author: U.S. Department of Energy
    Publication Date: 2014

    This summary from a Better Buildings Residential Network peer exchange call focused on how to identify stakeholders and map them to determine the best engagement strategy.

  2. Author: Jane Peters, Research Into Action, Inc.
    Publication Date: 2010

    This presentation covers the importance of collecting and evaluating program data, including data related to marketing efforts.

Publications

  1. Author: Lawrence Berkeley National Laboratory
    Publication Date: 2010

    This guide provides an assessment of various approaches to Marketing & Outreach for home energy efficiency improvements.

Webcasts

  1. Door-to-Door Outreach and Tracking Impacts
    Author: U.S. Department of Energy
    Publication Date: 2010
    Presentation, Media, Transcript

    This webcast discusses door-to-door campaigns and how to track the impacts of these campaigns.

Last Updated: 01/20/2016