This document highlights the best practices for using Myna. It provides advice on running experiments effectively and making sure you’re getting the most out of Myna.
Myna is best used for content that changes slowly and users that change interest slowly. It is perfect for testing user interface elements, marketing copy, and so on.
Myna works less well on content that changes rapidly. News items, for example, receive fluctuating interest that can confuse the Myna algorithm.
Create your variants with care. Myna can only do as well as the variants you give it. The fewer variants you give Myna, the more traffic it can push to each variant, and thus the faster it can find the variant that performs best.
For clean, conclusive experiments, only change one thing at a time. If you want to see how changing an image, some button text, and the colour of your header bar affects the conversion rate on a page, run the experiments separately and consecutively so that you can identify the specific change that makes your users more likely to convert.
Be bold with your experiments. Because Myna optimises on the fly, the rewards for being bold are far greater. Equally, the risks with sub-optimal changes are less severe. If your customers really don’t like the new colour scheme, for example, it will quickly be buried so you can pretend it never happened.
As you can add variants while an experiment is still running, you can react quickly to emerging patterns. If it seems, for example, that a UI element performs better when rendered in blue than in red, you might want to test out yellow, or drill down into different shades of blue.
Note that it can be useful to keep the original format of the element to use as a ‘control’.
You can stop an experiment at any time.
The conversion rate chart on the dashboard indicates the variant that Myna believes is the best and the confidence in that belief, so you always know which variant to choose when you finish.
Waiting until completion gives the most certain results, but if stopping an experiment while it is still running, you still know that it will have done the best possible job up to that time.
On the other hand, there is no need to stop an experiment. Experiments can run indefinitely - Myna will keep on optimising your pages in response to your users’ preferences.
Document, document, document
Remember to document the results of your experiments. We’re receiving many requests for additional metrics and reports from the information produced by Myna, so watch this space. If you have any specific suggestions, do let us know.
It’s also important to document the details of the experiments; since you’re using short, abbreviated names to refer to the variants it’s important to make sure that you know which variant refers to which UI element, URL, or text.
Server- vs Client-side integration
You can integrate with Myna on the server or the client.
- it is quick and easy to implement;
- calls to Myna don’t touch your web servers.
Server-side integration involves your server-side code making calls to Myna and incorporating Myna’s suggestions before content is sent to the browser. This has a number of advantages:
- page rendering times are reduced;
- you can use asynchronous code to mask time spent making API calls;
- you avoid exposing experiment IDs to your users;