Sitecore DMS Engagement Value - Part 1
When Sitecore introduced the concept of Engagement Value they created a valuable metric, especially for organizations that don’t collect revenue online. Determining where Engagement Value fits in the broader ecosystem of web and organizational metrics is a challenge for companies and their Sitecore implementation partners. We’ve investigated ideas for useful interpretations of Engagement Value and how it fits into the universe of website performance analytics.
Strengths & Weaknesses of Sitecore’s Standard Engagement Value
I discussed some of the advantages of Engagement Value (EV) during my Sitecore User Virtual Summit talk, but it’s also important to be aware of its flaws. EV is intended to quantify the level of interest and interaction that a user has with a website. By assigning EV scores to users as they meet certain predefined goals, we can begin to measure the user’s engagement with the website in numeric terms.
Looking at direct derivative metrics like Engagement Value per Visit provides insight into the performance of content on an ongoing basis. EV per Visit is a great metric for analyzing page as well as overall site performance. This all sounds great, but you might be asking, “Where are the flaws?”
Engagement Value and its direct derivations are effective in gauging the performance of content, but have limitations when measuring your audience behavior over time.
Take the following scenario:
A large content-producing foundation has a primary business objective of distributing content to a broad audience to form lasting relationships.
Bob is a site user who has just met 2 site goals by reading three policy reports and downloading an event transcript. For his efforts, the site assigns Bob 50 engagement points. If we set up a report to view the number of Engaged Users based on a threshold of 50 points, Bob will appear as one of them.
Now, Bob does not come back to the site for six months. When he does return, Bob receives 5 engagement points for his activities. This brings his total to 55 points. Herein lies the problem. Bob still appears to be engaged, but he is not!
A first limitation of Engagement Value is that there is no sophisticated way of tackling the problem of engagement over time. There is good news, however. We have ways to transform the standard Engagement Value and start answering questions like ‚ “How many successful relationships am I currently maintaining?” or, in the case of commercial entities, “How many warms leads do I have today?” Tracking these values over time can begin to describe the successes and shortcomings of changes to the website and various inbound channels.
Creating Meaningful Metrics Based on Engagement Value
We can begin to answer these questions with Decaying Engagement Value (DEV). Using a decay concept on Engagement Value allows us to consider the timing of events when assessing the importance of engagement. This is a calculated metric that we can set up in our Business Intelligence tool by using the concept of half-life. Half-life indicates the amount of time it takes for the original value to be reduced by half. For example, we can think of subscribing to a newsletter as an event worth 50 points on day 1. The half-life is the number of days until that same event is worth 25 points. There are several mathematical models used to calculate half-life, this being one of them:
DEV = EV (0.5)^T/Th
- Engagement Value - Points assigned to the completed task/goal
- T - Duration in days since the event was completed
- Th – Half-life of engagement value on the current site (30 days may be appropriate for our example)
The advantage of using a half-life formula (as opposed to a linear decay model) is that the time penalty is most severe upfront, but then takes longer for the event to have no impact. With a linear decay, the event is dismissed entirely at a certain point. Therefore, when looking at a user event, the change in engagement between days 80 and 90 is less impactful than the difference between days 1 and 10.
Decaying Engagement Value is calculated on a per event basis, as EV is captured for each event. This means that each event has a date associated with it, and the value associated with each event will decay at its own pace. This information can then be aggregated on a user level at the time of creating the report.
Measuring Your Websites Engaged Visitors
Using the Decayed Engagement Value metric to determine Engaged Users, graphed in a month-over-month or week-over-week representation allows fantastic topline insight into a company’s overall efforts around their website. I say “overall” because this metric is a bit broader than just the website performance, and also serves as an indirect way of measuring the effectiveness of inbound campaign efforts.
This may sound like a bold claim, but I believe it is justified. Engaged Users shows a point-in-time count of currently engaged site visitors. The goal of an organization is to grow this number over time. An upward trend in this number indicates one of several things:
- The website is doing a better job engaging users.
- Inbound channels are bringing more traffic to the site.
- Inbound channels are bringing more effectively targeted users to the site.
In a perfect world, all of these things are happening. While we have a topline metric to gauge all of those trends in aggregate, but we can still dig in deeper by further transforming the data.
Let’s look specifically at whether the website is doing a better job of engaging the users that it has.
We first calculate a metric based on the number of Engaged Users divided by the Total Unique Visitors. We look for an upward trend in this number, as it indicates that the website is doing a better job of engaging the users it has before. This could be attributed to website enhancements or campaign improvements bringing in a more targeted audience. The effectiveness of inbound campaigns can be controlled using some of the additional metrics outlined below.
Measuring the Effectiveness of Inbound Channels
Diving into inbound channel effectiveness, we can look at both Engaged Users by Source and per Total Unique Visitors. The former describes the overall productivity of an inbound channel, whereas the latter describes the overall engagement level of the audience captured by a specific channel. This last metric is particularly interesting as it allows us to infer the overall quality of traffic generated by the channel. Viewing this metric across channels allows us to control the ability to engage viewers and isolate improvements in specific channels. All of these metrics can help direct your marketing and communications resources to the most effective area, and trim spend on ineffective channels.
Engagement Value as an Indicator of Revenue
As an organization looking to increase content consumption and cultivate ongoing relationships with its users, the aforementioned metrics should help paint a comprehensive picture of those relationships. As a commercial organization needing to tie this user engagement back to its ability to drive revenue, there is a metric that can help make this final connection: Purchasers per Engaged Users.
Purchases by Engaged Users will show the correlation (or lack thereof) between Engaged Users and those who ultimately make a purchase. This metric helps with recalibrating Engagement Value scores to be better aligned with activities that ultimately lead to sales. For the more statistically inclined, running a linear regression with revenue as the dependent variable can give a more detailed view into how closely Engagement Value ties to generated revenue. The higher the R2 value of the regression (assuming the statistical significance of the EV variable), the better Engagement Value describes the variance in Revenue.
In Summary
These are just some of the derivations of Engagement Value that can help organizations leverage this new Sitecore metric. Engagement Value, like all other metrics, does not tell a complete story on its own. It is much more effective when viewed in conjunction with other measures. Different Business Intelligence tools, such as Tableau or Spotfire, can help to visualize this data and make the calculations necessary to tell the story behind the data.
Check back soon for Part 2 on how to extract this information from the Sitecore DMS analytics database and visualize it using Tableau.