Testing multiple elements on prototypes at once!
Picture yourself browsing the web for some information: Do you go out of your way to read the text on every Google page you open, look at all the graphics, and try to thoroughly understand what the page is about?
Well, we know what the answer is — No.. With so much of information around us all the time, our attention span has drastically decreased, not paying enough attention to what a particular web page wants to tell us.
In this post we will talk about multivariate testing, which is a A?B testing at its roots. But here’s the difference — A/B testing is testing on 2 versions of a design. Whereas, multivariate testing is testing on multiple versions of the design..
What is multivariate testing?
In a multivariate test, a design screen of a prototype is treated as a combination of elements (including headlines, images, buttons and text) that affect the conversion rate. Essentially, you decompose a design into distinct units and create variations of those units. For example, if your design page is composed of a headline, an image and accompanying text, then you would create variations for each of them. To illustrate the example, let’s assume you make the following variations:
- Headline: headline 1 and headline 2
- Text: text 1 and text 2
- Image: image 1 and image 2
The scenario above has three variables (headline, text and image), each with two versions. In a multivariate test, your objective is to see which combination of these versions achieves the highest conversion rate. By combinations, I mean one of the eight (2 × 2 × 2) versions of the Web page that we’ll come up with when we combine variations of the sections:
- Headline 1 + Text 1 + Image 1
- Headline 1 + Text 1 + Image 2
- Headline 1 + Text 2 + Image 1
- Headline 1 + Text 2 + Image 2
- Headline 2 + Text 1 + Image 1
- Headline 2 + Text 1 + Image 2
- Headline 2 + Text 2 + Image 1
- Headline 2 + Text 2 + Image 2
In multivariate testing, you split traffic between these eight different versions of the page and see which combination produces the highest conversion rate — just like in A/B testing, where you split traffic between two versions of a page.
Conducting a successful multivariate test
The best part about conducting a multivariate test on prototypes is saving a lot of effort and time spent on making multiple variations of the designs. Things move much faster when you have a low fidelity prototype to flush out versions. Having said that, lets talk about the steps performed in a multivariate test -
Step 1. Identify ‘a’ challenge
Identifying the challenge early in the project can be done only from user testing results. But for existing projects that require a revamp, its a notch easier.. Google Analytics is a great way to understand the challenges in the live website/app.
Lets take an example — Your page might have all the right ingredients: appropriate product name, product description, testimonials, awards, ratings and a prominent download link. Yet, only 30% of the visitors download the free software. So, why didn’t the remaining 70% of visitors download the software? In this case, fixing this leaky bucket is your challenge.
What when you have multiple challenges?
The project you are working at might have multiple goals. The best example of this situation would be a blog page where the challenge is to get more subscribers and also to increase visitor engagement (in terms of number of comments). In such cases, the best strategy is to tackle one challenge at a time.
Step 2. Structuring out a hypothesis
The next step is to make a list of hypotheses for the challenges you are facing. It is certainly tough to come up with exact reasons for your challenges, but there are resources to back your experience and gut feeling!
- Think like a visitor: Yes that is difficult — very difficult. The product is your baby and accepting that it has faults is not easy.. But, you have to be the strict father to your product. Step into the shoes of your visitors and try to complete the task keeping in mind that you have no background knowledge about the offering. Remember that your visitors don’t wake up in the morning saying, “Oh wow, this thing is just amazing!” You built the masterpiece and you are the best person to think of an alternate better solution, IF you accept that it can have a better solution.
- UX analytics data: (This is effective only if you have a live product) Another source for getting a list of improvement ideas is Google Analytics. For example, a lot of visitors may be arriving on your webpage by searching for keywords which you haven’t even thought about. In that case, your visitors may leave the website mistakenly thinking that your offer is not what they were searching for. Addressing such cases can increase the conversion rate.
- Usability testing on prototypes: Usability tests will always surprise you! Be it a low fidelity paper prototype or a high fidelity design prototype, conducting usability tests on them has always been the most useful way of building hypothesis about a challenge. If you don’t have a large budget for usability testing, try out affordable services such as CanvasFlip (Unlimited usability testing on prototypes)
Step 3. Design variations for test
Once your list of possible reasons for your challenges is ready, it is time to brainstorm design variations that might solve the problem. In this step, you come up with multiple different versions for the factors you came up with in the last step. Get all versions of the prototypes ready before you roll out the test. For the “Sign Up” case, for example, different versions will be:
- Form variations: Minimal form with just two fields; form not asking for email address; multi-step form; long form.
- Submit button variations: “Submit” or “Sign Up for Free” or “Instant Signup” or even “Sign Up Now!”
Step 4. Executing the test & analysing results
Now that your prototypes are ready, share different versions of the prototype with sections of your user base. Let them know that they are a part of the improvement process. It only gives them a sense of responsibility towards the betterment of the product. Once the users are done with using the prototype on CanvasFlip, the analytics is automatically generated.
The entire sessions of the users, the cumulative data on the drop-off rate on every screen, interaction per screen etc are recorded. Now you have to just pick the version with the best results and move ahead with it.
Driving a product to get all the love from the users is not easy, but tools can help you in the journey.. We built CanvasFlip with a similar motive in mind.. Give it a try and let us know how easy your multivariate testings become.
