Interpreting the results
The Myna dashboard updates in real-time to bring you up-to-the-minute statistics on your experiments.Here’s a guide to interpreting the charts.
The conversion rate chart
The conversion rate chart is the most important source of information in Myna. For every variant it shows you:
- the average conversion rate for each variant;
- the confidence in that average.
Average conversion rates
The average conversion rate tells the proportion of views of the variant that have resulted in a successful conversion. The bar is smaller for variants with a lower conversion rate:
and larger for variants with a higher rate:
In other words, the bigger the bar, the better the variant is performing. Myna automatically adjusts to show stronger variants more often than weaker ones.
Myna’s figure for the average conversion rate may not be the true conversion rate - the accuracy of the estimate depends on the amount of data Myna has gathered. For example, if you flipped a coin once and it came up heads you wouldn’t think the coin always comes up heads, whereas if you flipped the coin 1000 times and it came up with heads every time, you might begin to think it was a trick coin. For this reason Myna also shows a confidence region.
At the beginning of your experiment, when Myna has only gathered a small amount of data, the confidence region for each variant will be very large:
As the experiment progresses and Myna gathers more data, the confidence regions will shrink:
The confidence regions will become smaller and smaller as more people visit your site. Eventually, once the confidence regions are non-overlapping, the bar for the strongest variant will turn green:
At this point you can be certain that the green variant is the strongest.
The total conversions chart
The total conversions chart shows you:
- the total number of views;
- and the total number of conversions.
for each variant in your experiment.
The total number of conversions will always be less than or equal to the total number of views. This chart is primarily useful to see how much traffic your experiment is getting. It will also show you how Myna is adaptively displaying variants in response to conversions.
A variant with fewer views will have a shorter bar
than a variant with many views
Likewise a variant with fewer conversions will have a short total conversions bar
than a variant with more conversions
As noted above, a green bar indicates the optimal variant.
You can now refine your variants, or make a final decision.
If there’s no clear winner after a significant number of views, it may be that your customers have no particular preference for any of your variants. In this situation, we’d recommend you either add and test a new variant, or stick a pin in a board to pick one of the options.
The conversion rate charts for each experiment are scaled to fill the horizontal space available on the page. Because of this, different experimental charts are shown at different scales and you shouldn’t directly compare the widths of bars in different charts.
The true figures displayed on the charts, which can be compared across different experiments, are shown underneath the bar for each variant.