My secret sauce for coming up with great A/B test ideas
I am going to share one of my favorite ways to come up with a/b test ideas. I have been a part of countless ideation sessions, consulted many leaders on the topic, and discussed and learned many different approaches from experts across the country.
This post will apply to you if you are looking for a simple framework to come up with test ideas or need that one or two pieces of inspiration that can add fuel to your testing ideas.
The test ideas are so important! Too many companies jump past the idea itself and focus on shipping the test out, the analysis, and reporting. A standard A/B test in big companies can take between 45-100+ hours to build, run, and analyze. That is a lot of time and money, especially when what you spent all that time on was a lame idea.
Tip 1: Use a simple framework and do not over complicate the ideation process. Why? When the steps and requirements get too complicated, people lose interest. You want people jazzed and wanting to contribute, make it easy for them.
Tip 2: Focus on empathy in the “needs” and “problems” sections of my framework. Think about the consumer’s needs and work from there. For example….
Though there are software tools available to walk you through ways to come up with test strategies, many great ideas from my experience emerge from physically writing, visually connecting the dots and discussing with people.
For this exercise and simplicity’s sake, let’s focus on a single page test such as a homepage, cart, landing page, etc. The basic concept can be applied to a journey test where the a/b test can span multiple pages or channels.
Here are the steps:
Get a piece of paper out and orient it landscape. If you have a big whiteboard, even better! I love having the space to visualize and think.
Where are you going to test? Start with that at the top. Often the mandated organizational priorities drive the focus testing on certain pages or sections.
What is the business problem you are trying to solve? Examples are conversion rate, engagement, etc. Put this under the ‘where.’
Underneath the above, create 5 columns. At the top of the first one, write “segments” and underline it. Then “needs” in the next column, “problems,” “ideas,” and “top choices.”
Who are the segments? For example, you can have all visitors, logged in members, first time visitors, etc. I would limit to two or three. If you have all visitors and logged in members as the two segments you are considering testing for, draw a line across the page splitting the columns in half. In their respective segment boxes, write down the key things you know about them. Conversion rate, visitor volume, revenue impact, bounce rate, and other metrics as well as any insights you have from other sources of research such as focus groups, industry research, etc.
Why are they here and what are their needs? What pain are they trying to resolve? Think tactically as well as think about what I refer to as “deeper” needs. Example 1 is for the homepage for a non-profit charities site. A tactical need is the person is here to research to decide on whether to donate money to a cause through this organization. The deeper need is the person wants to feel like they are making an impact to the world by helping those who are suffering. Example 2 is for a kids fishing set in the kids toys category section of an ecommerce site. A tactical need would be that the person is here to find the right specs for his kid or a present for someone. A deeper need is to feel a sense of security knowing that his or her kid will be happy with the right fishing set. This is where my reference to empathy for Tip 2 comes into play. The visitor is at this page is there for a reason. If they do not have a need or pain, there will be no transaction. Try to dig deep and not just tactical. Many big wins come from striking that right message to address deeper pains than the tactical level.
What are the problems with the current experience that conflicts with fulfilling their needs? (these form your hypothesis) List out quantitative, qualitative, past test results, and intuition that leads to justifying why you think there are problems with the current experience. I like this part of the thinking. When you wrote down “deeper needs,” does the current experience address that? If you want someone to donate money to a cause because they feel empathy or pain for others and the current experience is very plain and boring, how does that make the visitor feel? Will they feel inspired that their contribution will make a big difference? You are selling your product or services in exchange for something. Whether that comes in the form of money, time, and/or resources, if the consumer does not feel the exchange benefits them, there will be no sale.
What are your ideas for solutions? For each of the problems, you can come up with multiple ideas. Draw a line from a problem across to the ideas column and start coming up with ideas. Next to each idea, write down the level of development effort, legal/brand issues, and boldness. There are typically the basic three you to consider though different organizations have different factors to consider.
What are your top ideas for solutions? You should have a list of ideas now along with possible red flags! Choose which ones you feel are realistic to pull off and will make the most impact to the business problem you are trying to solve.
Below is an example of how it would look like on a piece of paper. A shout out to The Dr Bell Leadership program :). Use a normal pen so you can squeeze in a lot more content. Again, a big whiteboard is great, especially if you are going through this with others.
Regardless of how the test turns out, going through these steps should give you insights and why you decided to test what you did. The more you do this, the easier and faster you will get through this exercise. Many companies spend far too little thinking on the ideation part of the entire test life cycle and they limit the efficacy of their tests because of poor planning.
Remember, keep the framework simple and spend some time thinking through how your tests can change how consumers feel.