How does Myna work? Through the power of science!
How does Myna really work? Myna uses Multi-Armed Bandit (MAB) algorithms to optimise experiments. The precise implementation is based on recent published research combined with our own brand of secret sauce - it is considerably more effective than algorithms such as epsilon greedy that are often discussed in blog posts about MAB testing. For more information, see Bandit Algorithms and You.
Seasonal and weekly effects Experiments should be running long enough for reasonably short-term variations to be included in the results. And as Myna continuously optimises your site, you can actually take advantage of longer-term, seasonal variations by keeping your experiments running. So scheme that is more effective in spring but less popular in autumn, for example, will naturally self-optimise over the year.
Is it possible to integrate with my analytics platform of choice? It is possible to integrate Myna feedback with analytics programs - see the page for the relevant client library for more information.
Don’t Myna’s optimizations risk bias in the results of my experiment? Myna constantly balances a trade-off between exploiting good variants, and exploring other variants to maintain fairness. Good variants are always found, even if they happen to perform poorly in the opening stages of an experiment. For more information, see Bandit algorithms and you.
Won’t this confuse visitors, to see a different page every time they load? It depends on how big the differences are between your variants. For small differences there should be no problems. For larger differences, we recommend you clamp users to a single variant using client code. The
What makes an experiment statistically significant? Numbers. More numbers. The more views, the more conversions, the better the experiment. Myna makes this easy for you by clearly displaying the confidence interval of your experiments. When Myna is confident that the results of the experiment are conclusive, the best variant turns green to indicate that you can end the experiment. For more information, see Interpreting the results.
Do you change my website or do I? At present, the Myna testing process involves you making amendments to the code of your website. The process is quite straightforward, and we have guides and code samples in various languages. We are working on making Myna easier to use by non-technical people.
So I need to be an advanced developer to use it? Not particularly. You need to be comfortable with editing the code of your website, and understand the basic principles of coding. We’ve provided code samples to make the process as painless as possible.
Can Myna access my data? No. Myna only reacts to the experiment requests you send to it; so unless you use super-secret passwords to name your experiments, it’s all entirely impersonal.
Are you planning to offer a more user-friendly interface? Yes! It’s coming - watch this space…
So the final decision is automated? Yes! Isn’t that awesome! Because Myna automatically (automagically?) optimises the display of your variants, at the end of the experiment it will already be displaying the ‘winning’ variant. You’ll be sent an alert notifying you that this is the case, so you can either end the experiment, or add new variants as you see fit.
What happens if there isn’t a clear winner? At the moment, nothing. You’ll still have to decide between your options. But we are planning to implement notifications that you can set up for time or a set number of conversion counts so that you can review and make any changes.
Isn’t optimise spelt with a z? We’re from the UK, so we use British spellings.
Any more questions?
If you have any more questions, please don’t hesitate to contact us.