How the Landscape of Market Research is Changing Faster, Better Insights with AI
Kristyn Corrigan, Applid Marketing Science
kHUB post date: July 29, 2020
Originally presented: November 5, 2019 (PDMA 2019 Annual Conference)
Read time: 7 minutes
Download the Handout
Machine learning is an increasingly prevalent term in today’s market research landscape. But what is it and why is it so important?
The ultimate measure of successful research in a mature and well-understood category hinges on its ability to uncover new, game-changing insights. Traditional market research can be expensive and time consuming, and in today’s hypercompetitive digital landscape, it can be harder than ever to find truly new, actionable insights. Now, more than ever, product developers expect to quickly and easily discover new insights on smaller budgets and within shorter time frames. But how can researchers maximize the chances of uncovering new, breakthrough insights within those constraints? The impactful insights that product managers and marketers are looking for can now be discovered without collecting primary research data. Machine learning and AI, when coupled with professional analysts, expedites the discovery of customer needs from user-generated content (UGC) and identifies “needle in the haystack” insights that are either infrequently mentioned or overlooked completely in more traditional qualitative research.
Mining user-generated content
Machine learning algorithms can utilize a variety of content types and thus it’s critical to identify the best sources of user-generated content for your project. In some cases, useful content exists on product review sites, social media pages, customer forums, or in blog posts. In others, it exists in call center data, transcripts of customer interviews from prior studies, or answers to open-ended survey questions. Regardless of the source, customers are discussing your brand and sharing their opinions and stories. These data are a goldmine for machine learning analyses.
The benefits of machine learning
While the sheer amount of customer data may seem overwhelming, the good news is the machine is able to do the heavy lifting.
Key advantages of the machine learning method include:
- Faster: AI is faster than traditional research because of the machine’s high processing power. The machine can process hundreds of thousands of records of data in a matter of minutes.
- Better: Machine-based analysis is able to overcome human research bias. The machine has no hidden agenda and does not heed to corporate politics. Unlike humans, it does not come in with preconceived bias and is completely objective in what it finds.
- Cheaper: UGC is virtually free, with the only nominal cost being the data scraping tools used during the process. AI cuts out the costs used in traditional market research, such as recruiting fees and incentive fees.
- Complete: Machine learning rapidly scans hundreds of thousands of data, ensuring all insights gathered are accurate and complete. Traditional research can be limited in collecting respondent data based on time, sample size, feasibility and cost.
- Easier: The machine streamlines the process of data reduction, pulling significant insights from massive data sets.
- Relevant: Customers often post about their experiences right after key moments of truth. These polarized experiences are what frequently give you the informative insights.
Developing the machine learning algorithm
In collaboration with researchers at MIT, AMS developed a proprietary AI methodology called ACE™ (Automated Content Evaluator). Link: https://ams-insights.com/insights-for-business-markets/big-data-analysis/ This tool uses convolutional neural networks – a type of supervised machine learning –to dramatically reduce the time and effort required to gather a comprehensive set of customer insights in a category. While many tools summarize key themes and keywords mentioned in big data, ACE™ dives deeper and identifies “pearls” of insight that traditional market research methods like focus groups and interviews might overlook.
Working with researchers at MIT, the effectiveness of the tool was proven by comparing the machine’s results with the results of traditional research studies. For example, in the oral care category, the tool uncovered nearly every unique customer need that was identified with traditional research. Furthermore, machine learning can identify insights that are unlikely to appear in a random sample of user generated content of equal size. The machine unearths critical insights that are unlikely to surface through random sampling. Time and again the technique has proven itself. In cases where the data already exists, ACETM consistently gathers fascinating insights for innovation faster and cheaper than traditional market research methods.
Successful applications of machine learning
Machine learning has been used by both B2B and B2C companies to gather a complete database of customer insights in only a few weeks. Successful machine learning research projects have been completed in a myriad of categories, including small kitchen appliances, skin and hair-care products, prepared foods, oral care products, drug-device combination products, and even snowplows. Even those in mature categories have discovered new insights that were not highlighted in prior research.
Further, machine learning algorithms allow firms to examine product and service adjacencies in-depth. Multiple adjacencies can be explored in a single research study, contributing to the machine’s agility, speed, and efficiency.
The market for kitchen blenders is highly competitive. Characterized by near constant product innovation, consumers face an overwhelming number of choices, each boasting the latest features designed to make their lives easier in the kitchen. Working with researchers at MIT, we used machine learning to identify a full set of customer wants and needs relating to blenders. We identified 97 unique consumer wants and needs related to blenders, covering 34 different topic areas. Topics included cleanliness, durability, ease of use, portability, safety, and the terms of warranty. The needs identified were highly detailed and actionable. For example, the needs detailed exactly the types of foods consumers are looking to pulverize and which types of foods blenders currently on the market struggle to grind up. The machine-learning approach accomplished all of this at a fraction of the time and cost of traditional research methods.
Additional helpful analyses
- The ability to use quotations to understand the relatability of the needs
- Sentiment analysis
There are many additional benefits that are evolving from the use of AI in analyzing customer experiences. One benefit to this method is that each customer need can be traced directly back to a verbatim, or quotation, detailing a customer’s experience. These rich verbatim can then be used as input to a quantitative survey where customers are asked to rate the importance and relatability of those verbatims. Negative verbatims that customers find relatable, for example, can often be areas ripe for product innovation. Using these illustrative quotes when presenting key research findings can also help clarify your findings and engage key stakeholders. Additionally, incorporating sentiment analysis can add further depth and richness to those customer experience profiles.
About the Author
Kristyn Corrigan is a principal at Applied Marketing Science where she leads the Insights for Innovation Practice. In her nearly fifteen years of consulting experience, she has helped dozens of companies worldwide to use customer insights to create more successful products, services and customer experiences. Kristyn is most energized when she’s helping clients uncover meaningful customer insights that change the way they think and that create real business impact.