Partner & COO
Experiment Designer
Experiment Designer
Let’s dive into the current state of synthetic testing. We’ll explore what it is, whether it actually works, the different types, available tools, and real-world use cases. We’ll look to the future and discuss where this technology is headed. And we’ll offer some practical advice based on our work with clients, answering key questions like how to get started and experiment with synthetic testing.
Let’s begin with a quick definition: synthetic testing uses various technologies, like AI, to mirror demographic, psychographic, and behavioral profiles, allowing rapid “consumer testing” and insights across a product’s entire life cycle.
Right now, there’s a bit of a buzzword bonanza; a lot of different startups are coining their own terms, there’s a lot of papers out there talking about things like synthetic data, multi-agents, custom GPTs, synthetic panels and so forth. So before we dive in, let’s dive into the different types of synthetic testing, listed according to increasing complexity, starting from the easiest to implement.
Custom GPTs: Customized Chat GPT instructed to meet specific tasks and mimic real-world user behavior. For example, pretending it’s a procurement manager in a large FMCG company and asking for feedback on a pitch.
Synthetic Users or AI-Simulated Ethnographic Research: LLM Models prompted to mimic specific personas or a small panel to simulate qualitative ethnographic data, such as in-depth interviews.
Synthetic Panel or AI-Generated Quant-Data: Synthetic data replicating quant research results using representative demographic mirrors, like age, gender, and income (e.g., a pricing research survey).
Agent-based bots: Intelligent system programmed to perform tasks, make decisions, and interact with their environment just like humans do. (E.g., Agent “EPC Contractor”)
Digital twins: Precise digital replicas of existing users or personas, created using extensive data on their behaviors and reactions.
It’s important to keep in mind that this is a thriving and rapidly evolving ecosystem. We’ve tested various tools over the past year and noticed significant improvements in this short amount of time.
The big question clients always ask is, “Does this actually work?” It’s a simple question but requires a complex answer.
Scientific studies show that language models can make relevant predictions about human behavior, especially in specific, experiment-driven scenarios. For instance, a 2022 study used GPT-3.5 to analyze buying behavior for laptops at different price points. The results matched basic economic principles, like demand decreasing as price increases. This highlights that GPT-based analyses can at least complement real-world ones. A GPT-based study with over 10,800 responses takes just half an hour and is very cost-effective, compared to the weeks needed for traditional methods. This suggests that language models can quickly and cheaply provide valuable initial insights to guide more detailed studies later.
Some clients express disappointment with synthetic testing, finding it falls short compared to traditional consumer research and provides vague insights. This often stems from unrealistic expectations; If you expect synthetic testing to exactly match traditional methods, you’ll likely be dissatisfied. It’s crucial to frame these tools correctly and recognize their value when used appropriately, despite their pitfalls.
Responses may lack the depth and specificity needed for detailed analysis and decision-making, with outputs perceived as poor or too generalistic.
Synthetic Users may rely on training from outdated data, limiting relevance for forecasting current and future trends.
Poorly crafted prompts lead to vague or irrelevant outputs.
Agents struggle in environments requiring responses to multiple stimuli (e.g. beyond just text).
Simulating intricate human and social dynamics within a synthetic environment is computationally demanding and may lead to inaccurate results.
Synthetic personas may be exposed to security vulnerabilities and breaches, where sensitive or confidential information may be disclosed to third parties.
To drive a proper buy vs build discussion, we need to dive into the main attributes of synthetic testing and what is driving the success and quality of the outcomes that you’re going to be getting out of the synthetic testing tools.
In the future, we anticipate a new type of synthetic agents – specifically, Living Audiences – where consumer personas are not just static and reactive, but live and proactive. Living Audiences will offer a 360-degree view of consumer behavior, fueled by live data from external and internal sources, and by insights gleaned through synthetic testing interactions.
When considering the architecture for a living audience infrastructure, you need to integrate data, human feedback, and advanced AI layers to create an adaptive and dynamic system. From the data input perspective, this includes social listening, surveys, current CRM interactions, and e-commerce data, along with static consumer and brand knowledge. Additionally, targeted real consumer research will be essential to continuously fine-tune and tweak your living audiences or synthetic testing tools within your organization.
We see this evolving into a more autonomous agent system, where multi-agent interactions closely mimic real consumer behavior when combined with human feedback loops. This ensures you don’t end up in an AI echo chamber, but instead, receive genuine human inputs.
Finally, all of this needs to integrate into a solid user interface, and you must identify the actual use cases. Do you want to use synthetic or living audience feedback for co-creating new products? Perhaps you need continuous segmentation for different brands? For example, large CPG companies often have broad segmentation, but with living audiences, you can tailor this down to specific markets and brands with much more granularity. You could also run scenario analyses, such as predicting the impact on your portfolio if you launch four new products or discontinue three others.
In essence, the business outcome of Living Audiences is higher hitrates, faster go-to-market, and better products brought to market.
Across the end-to-end product development cycle, synthetic data at its current state is already adding a lot of value. For example within ideation, you can use different types of synthetic testing to inform, gather initial feedback on early concepts, have it help you prioritize between hundreds of concepts, and from there on continue with more traditional research methods, for example, running a basis analysis, doing some qualitative interviews to do final tweaking and get the human validation. And as we start moving towards Living Audiences, we’ll for example be able to do predictive concepting, autonomous iteration of concepts and conducting real-time product/market fit.
In order to get the most out of synthetic testing, you need to keep the bigger picture in mind and understand the future potential and current limitations of the technology – and remember the speed at which this tech is improving. This is typical of the innovation curve: high early expectations can lead to initial disappointment.
Getting started is all about setting the vision, managing expectations and building towards early proof points.
With the right expectations and strategic approach, you can effectively leverage synthetic testing and living audiences to unlock their full value.
Want to discover how you might use synthetic testing to get real-time insights from AI-modeled customers, and test concepts in hours, not weeks? Let’s talk!
