Designing experiments

Here’s how to design an experiment for Myna.

In order to create a successful and instructive experiment, you need to:

  • identify a specific, measurable goal
  • choose the part of the page you want to test
  • decide on the variants you want to try

Identify your goal

You’ll likely have an overall goal for your page: increasing sales, signups, or page views for your content (which may also lead to increased sales). The first thing to do is to determine the specific actions your visitors perform that lead to this goal:

Possible options include:

  • the visitor signs up for your free trial
  • the visitor follows a link deeper into your site
  • the visitor finishes your check-out process.

The important thing is to identify a specific interaction with the page: clicking a link, ticking a box, or playing a video. This will enable you to send Myna the ‘reward’ that fuels the machine learning process.

You can have more than one goal per experiment, but we recommend starting with just one.

Example:

Monique Myna has started a blog on antique typewriters. To increase her readership and make a little money from adverts, she wants to increase signups to her mailing list. In this situation, the goal interaction is the user click on the ‘Sign Up’ button.

A goal on a web page

Choose the part of the page to test

Many subtle tweaks to your page design can have a dramatic effect on the goals you defined above. Sensible things to test include:

  • images - particularly product or hero images
  • text - including product descriptions and/or labels on buttons or links
  • colours - including highlight colours that call out parts of the page
  • layout - including positioning above or below the fold, location of buttons and forms

When deciding what to change, you may want to think about common concerns such as usability, consistency, or branding. However, almost any change can be worth trying if you are unsure where to begin.

It is possible to run multiple experiments in parallel on a page. However, this can lead to unexpected results unless you know what you are doing. To begin with, we recommend you only test one thing on a page at a time. See the section on best practices for tips on running multiple experiments in parallel.

Example:

Monique lists the possible changes she might make:

  • the position of the sign-up form
  • the colour of the button
  • the label on the button
  • the presence/absence of button (i.e. enter email and go)
  • the hero image on the page

She decides to start by testing the hero image, as this is the most obvious way of setting the tone of the page.

Possibile variables to alter on the page

Decide on the variants you want to test

This is where you decide what actual changes you want to try. These can be wildly dissimilar or have barely perceptible differences.

You can have as many variants in an experiment as you like - if you have four possible options, you can test them all at once. Note, though, that experiments with more variants take longer to run. We recommend you limit experiments to two to four variants in most cases.

Example:

Monique produces three potential hero images and crops and tints them appropriately to look good on the page.

Possibile variants of the hero image

Summary

At the end of the design process, you should have:

  • one simple, short-term goal and reward scenario
  • one variable element to test
  • two or more variants of that variable

Once you have these, you’re ready to set things up in Myna.