Interpreting the results

The Myna dashboard updates in real-time to bring you up-to-the-minute statistics on your experiments.Here’s a guide to interpreting the charts.

The conversion rate chart

The conversion rate chart is the most important source of information in Myna. For every variant it shows you:

  1. the average conversion rate for each variant;
  2. the confidence in that average.

Conversion rate chart

Average conversion rates

The average conversion rate tells the proportion of views of the variant that have resulted in a successful conversion. The bar is smaller for variants with a lower conversion rate:

Conversion rate chart showing a low rate

and larger for variants with a higher rate:

Conversion rate chart showing a high rate

In other words, the bigger the bar, the better the variant is performing. Myna automatically adjusts to show stronger variants more often than weaker ones.

Confidence regions

Myna’s figure for the average conversion rate may not be the true conversion rate - the accuracy of the estimate depends on the amount of data Myna has gathered. For example, if you flipped a coin once and it came up heads you wouldn’t think the coin always comes up heads, whereas if you flipped the coin 1000 times and it came up with heads every time, you might begin to think it was a trick coin. For this reason Myna also shows a confidence region.

At the beginning of your experiment, when Myna has only gathered a small amount of data, the confidence region for each variant will be very large:

Conversion rate chart showing a low confidence estimate

As the experiment progresses and Myna gathers more data, the confidence regions will shrink:

Conversion rate chart showing a high confidence estimate

The confidence regions will become smaller and smaller as more people visit your site. Eventually, once the confidence regions are non-overlapping, the bar for the strongest variant will turn green:

Conversion rate chart showing a definite best choice

At this point you can be certain that the green variant is the strongest.

The total conversions chart

The total conversions chart shows you:

  1. the total number of views;
  2. and the total number of conversions.

for each variant in your experiment.

Total conversions chart

The total number of conversions will always be less than or equal to the total number of views. This chart is primarily useful to see how much traffic your experiment is getting. It will also show you how Myna is adaptively displaying variants in response to conversions.

A variant with fewer views will have a shorter bar

Total conversions chart with fewer views

than a variant with many views

Total conversions chart with many views

Likewise a variant with fewer conversions will have a short total conversions bar

Total conversions chart with fewer conversions

than a variant with more conversions

Total conversions chart with many conversions

Confirming completion

As noted above, a green bar indicates the optimal variant.

Conversion rate chart showing a definite best choice

You can now refine your variants, or make a final decision.

If there’s no clear winner after a significant number of views, it may be that your customers have no particular preference for any of your variants. In this situation, we’d recommend you either add and test a new variant, or stick a pin in a board to pick one of the options.

Comparing charts

The conversion rate charts for each experiment are scaled to fill the horizontal space available on the page. Because of this, different experimental charts are shown at different scales and you shouldn’t directly compare the widths of bars in different charts.

The true figures displayed on the charts, which can be compared across different experiments, are shown underneath the bar for each variant.