Your Idea Counts: Idea Valuation (aka Reporting or Analytics)

This blog series publication was originally created in Jive Works and is faithfully replicated here as a convenience to our users.

Brandy Robert, Senior Manager, Proactive Service Delivery, Oracle Corporation and Rob Shapiro, Senior Director, Customer Service Experience, Oracle Corporation, have teamed up to implement ideas in My Oracle Support Community. The Your Idea Counts! series of blogs was co-authored by them and will deep-dive in to topics such as why idea generation is important today; ways to capture ideas; user and business impact; changing company culture to rally around ideas; and, of course, measuring idea ROI’s, KPI’s and other intangibles.

Do you really have the foundation for why ideas count and The Thinking Model? If not, review Your Idea Counts: How to successfully implement ideas in a global customer community Do you really understand why an engagement model is so important? If not, review Your Idea Counts: The Importance of an Engagement Model Have you put together an implementation plan? This is really important, so if you did not, please review Your Ideas Count: Implementation Planning and write one. Finally, did you consider gamfication, real-world rewards, ways to capture ideas and an established process (internal documentation) to lay out definitions, roles and responsibilities? Review Your Idea Counts – Go Live to make sure you are as complete as possible before you go live.

Idea Valuation (a.k.a. Reporting or Analytics) 

Finally, it’s time to talk about idea valuation (also known as reporting or analytics). We need to have concrete evidence to show the value of having ideas and that it is working to support the greater goals of the company and its customers. The problem, as with many other aspects of communities, is that it’s not the easiest to prove. We won’t even try to pretend we know all the answers on how to prove value, but there are a few things we can share and we hope you will heavily comment in this blog so we can, in turn, learn from you!

Proving that ideas are resonating, that they are important and that they are relevant is an uphill climb. It goes without question that an executive is going to ask the tough questions and we had better be prepared to tell a story. As anyone would think, there are basic metrics; metrics we might step up or elevate to be called Key Performance Indicators (KPI’s); and, metrics that are modeled to demonstrate a Return On Investment (ROI’s). Surveys, both external and internal should also be considered to either fill gaps of knowledge or acquire data not otherwise attainable. We assume this is already in place for you, so we are not going to address it here. If you don’t have these, consider creating a customer success survey to specifically capture customer feedback.

Basic Metrics 

We won’t spend a lot of time here because the basic metrics should be obvious. At a minimum you should track ideas created, commented on, voted on, liked, shared and bookmarked. Used in conjunction with user data, geographical data, and any other meta data that has attributes which can be reported on will give you a fundamental dossier of fundamental activity. There is no attempt to minimize what this can offer you as these can and will tell you part of the story, it’s just that they are not in the class of “why do I exist?” or “why should I invest?” categories.

Key Performance Indicators (KPI’s) 

Once we get past basic metrics, we have to grapple with the next level of complication to assess if a measurement we want to gauge falls in the category of really important -or- is so important that it will actually tell a part of the story that demonstrates a return on investment. On one level this will be subjective when considering your company’s goals and objectives. On another level, we might argue that there is important and then there is important!

For us, which may or may not agree with your thinking, we like to group the following as KPI’s:

  • Percentage of each stage. Oh, it’s really tempting to demote this to a basic metric but consider some of the things this might tell you. For instance, would a high % of a stage called Not Considering represent a disconnect between your company and the user base? Or, would a continual high % of Active or In Progress indicate that your product or service has a quality problem? A wrong market target problem? A niche problem? Your stages are a transparent roadmap to your users, based on their feedback, and as such is a KPI.
  • Idea coverage. This is an interesting one and can be used in 2 ways. When you have many communities it can be a % of how many communities are using ideas against the actual number of communities you have. This can be especially important and relevant when you have 100’s of communities (as we do). It’s analogous to a community measurement we pay attention to, namely community coverage to gauge what part of 100% of product/service are represented by one or more communities. It may not always be appropriate or warranted to have ideas in a specified community, so the advice here is that the baseline has to be adjusted to not count those as a part of this measurement.
  • Percentage of committed ideas. If you take all the ideas that have a status which represents a firm commitment (for instance, Implemented + In Progress + Under Review) and divide this by the total ideas you will have a number that can mean you are successfully (or are not) engaging your customers in the future of your product or service. There is also a “read in between the lines” factor with this measurement because it could also represent whether or not having ideas on the product or service is resonating or attracting your customers to use it.
  • Age of active ideas. How long is an idea in an active stage such as Active, For Future Consideration and however else you define a stage that is open without commitment. To make the most sense out of this measurement, you would typically want to bucket each active stage with some period specificity such as < 1 month, < 3 months, < 1 year and > 1 year (just an example). In some respects, the age of an active idea is also an expectation you are setting with your customer which obviously could be good or bad relative to your ability to deliver, but also relative to the cadence of your releases. This is surely something not to ignore in either case of good or bad.
  • Selected views. We often pooh-pooh the concept of reporting views as anything more than basic, but consider some aspects of what views could be telling you. For instance, looking at delivered stages (Delivered + Partially Delivered), over a period of time this might show you an important trend and one that is influenced or impacted by your release cadence or your active participation with your customers — a potential diamond in the rough.  Things to look for and consider would be cases where there is a high # of views, but maybe a low corresponding number of votes or comments.  This could indicate that the idea is not resonating with the users, though something triggered them to view it, so this could require further investigation or thought.

Return On Investment (ROI) 

  • ER efficiency. In the beginning, this is arguably the most important ROI. What you need to measure this is a snapshot of your tracking system(s) as a baseline to measure. For instance, if customers log a ticket (we’ll use this generic term to describe the technical support call) to express and document their idea that then transforms or is bridged to a tracking system where you keep “enhancement requests” (ERs) then you have the baseline measurement for 2 types of ER efficiency – one each for the ticket and enhancement request itself. Going forward, you will continually compare the creation of ideas to the # of tickets which if successful will show # ideas created going up and # of tickets (cases) and # enhancement requests going down. The goal with these trends will be to see # tickets reduced to 0 and the # enhancement requests going down to the number of enhancement requests you are actually committing to be done (i.e. no more noise, no more duplication, no more historical backlog!).
  • Ideas implemented. Here, you will measure what % of the ideas created were implemented, i.e. Ideas implemented % = (Delivered + Partially Implemented ideas) / total ideas. A successful ideas implementation will show a high percentage but what that high percentage is needs to be determined for your business relative to the product or service. For some, it could be 90% or more. For others it could be 75% and yet others it could be 50%. The point here is that you will need to put a stake in the ground (which, of course, could change as the business, products, services and customers change) on the advice of executive/senior management. This stake in the ground is also an inference for executive buy-in.
  • Releases implemented. While this could be a KPI, we position it here in ROI-land because initial implementation of ideas might need the support! Once you establish that your releases regularly incorporate ideas in a mainstream capacity, demoting this to a KPI might be ok. Obviously this measurement is: Release implemented % = (Delivered + Partially Implemented ideas) / total # release enhancements. This could also be called or seen as time to market.
  • Increased # users because of ideas or since ideas were implemented. This might need some fancy footwork or a survey. Inspect the user growth side-by-side with idea participation (creates, votes, comments, likes, etc). Your platform will need to be able to track by user and participation. Assuming you can get that, does the growth per user represent the increase in idea participation? It’s a tricky correlation and you may be better off having a survey asking about coming to your community because of the ideas implemented there. If you can prove it on the platform or with a survey, you definitely want this in your bag of ROI tricks.
  • License or subscription renewals. Optimistically you will show growth of license or subscription renewals, but there is also cancellations to consider as well. The thought here is to measure this as a % of ideas, although which part of ideas will be up to you. We can see looking at this based on created, active, delivered and all three of these as viable.

We hope that this list of KPIs and ROIs will get you thinking about ways to have more insight into the idea exchange and its success. The list is not exhaustive and will probably evolve over time as we become more understanding of the model, the process, and the customer response.  Above all, we hope that these measures will give some insight into the community engagement.  It is not until the conversation begins do we get a clear and fruitful understanding into our product and service experience. Until then, we can only partially understand and know what the market demands and understand how the market is changing.  Something that would really help is a “sentiment engine” to harvest and translate what is happening in the idea discussion. With a tool such as this, you could probably measure some level of positive or negative sentiment from within the comments.  These have been developed for various social channels but do not appear to be mature enough to bet on the ranch. However, it does not mean they can’t provide valuable insight into topics that are important to the customers and users.  They could even help to trigger further engagement by the product or service owner.

Whichever way you view measures of success, you won’t have anything to measure if you don’t get started! We hope that this blog series has given you some ideas of your own as to how you can align more closely with your customer base. We truly believe that starting conversations that appear to the collective audience will truly help to drive better product affinity and uptake. We hope you have learned from us, and in turn we would like to learn from you.  Our list is hardly comprehensive and this is where you come in. So, in being consistent with our approach in this blog series:

Final Action For You: Please join us to further develop how to measure ideas by adding your comments here! We are certain that we can all learn from each other and if enough comments are made we will be happy to collate everything in to one more blog in this series to reflect your feedback!