Posted by James Root
Improving search relevance is both an art and a science. Cirrus10 is able to optimize Endeca to consistently yield quantifiable improvements to search results with the greatest possible relevancy using input from merchants, customers, and our own secret sauce. In this installment of our blog, we provide a peek at the science of how we approach measuring the relevance of search results in the context of e-commerce product searches. The art, of course, will remain our little secret.
To begin – what is relevance?
Relevance in eCommerce denotes how well a retrieved product or set of products meets the need or goal of the customer. Relevance may include concerns such as color, size, timeliness or price of the result.
Increase the relevance of your ecommerce search.
Let’s take the example of someone searching for red Valentino shoes at Neiman Marcus. Are they looking for Valentino shoes in the color red? Or are they looking for RED Valentino shoes the brand? Or are they looking for red RED Valentino shoes (i.e. both)? How do you determine which results are most relevant and what to present to the customer in this situation? And then how do you measure the impact of that result set vs another?
There are many ways to measure search accuracy such as A/B testing and analyzing your search analytics. Every time you change your search algorithm you probably use one of these methods to gauge the success of your changes. One way we have found to be particularly important in measuring relevancy is to simply ask your various constituents what they think.
First, ask the merchant team. They will provide their perspective on what they want to sell and how they want to sell it.
Merchants offer the following:
- Internal view of what is important, what’s working and what’s not
- Gives you a quick idea of where to focus your tuning efforts
- Merchandising vocabulary
Second, ask the customer. Or a close approximation of your customers through crowdsourcing. Ask them to evaluate the results from your top 100 and bottom 100 performing search terms. Since every eCommerce site is unique there are no hard and fast rules about what patterns should emerge when looking at the metrics. It’s up to you to identify the patterns and insights that apply in your case. (When you are working with Cirrus10 we can also help as we have a very broad range of experience looking at patterns in search).
Customers offer the following:
- Large scale
- Insight into what users actually want and how they are behaving
- Simplified vocabulary
But because human judgment is subjective, at the end of the day our goal should be to get individual biases to wash out. By combining the merchant perspective with the customer perspective we can bring visibility to what is truly relevant.
How do we do this quantitatively? We compute metrics from the scored results using Normalized Discounted Cumulative Gain, a particularly insightful algorithm for comparing disparate sets of results. The basic steps are as follows:
- Step 1 – Give sets of results to your merchants and customers for judging, have them give each product in each position a score from least relevant to most relevant – 0 being least, 2 being most.
- Step 2 – Sum up the scores for each result – the Cumulative Gain.
- Step 3 – Calculate the weighting for each position – the discounted part. We want to measure out position bias by weighting the scores by the position.
- Step 4 – Normalize the values. Normalizing allows us to compare the scores across the many different queries.
- Step 5 – Average across all the queries and voila, a baseline measurement that gives you an overall score on an algorithm and allows comparison between two algorithms, which then gives you directional guidance on how to further improve your search results
Once a baseline has been established there are a number of really cool ways to improve Endeca’s search algorithms with custom code development. How this is approached will be based on each online retailer’s unique product set, merchant naming system and product attributes, tags and categories. This is the art and it is one of Cirrus10’s core competencies.
As a way to get started with both the science and art of improving search relevance, Cirrus10 offers a guaranteed ROI with our basic Oracle Endeca tuning program. Better conversion, larger shopping cart size and higher customer satisfaction are all within reach. If you would like to get started earning more money from your Oracle Endeca Commerce Platform, call Cirrus10 at 866.996.9985.
