How to create an A/B test?

An A/B test is a way to split your message deliverability into testing groups that will help you decide which version performs better. We have a dedicated section just for A/B tests from which you can create, manage, and send them. We will present your variants and the winning message in an intuitive way. You also create them using our state-of-the-art editor, which makes creating beautiful messages a breeze.

How do I create an A/B test

We offer you two types of A/B test. You can either test the subject line or the content of the messages.

How do I test a subject line?

In order to create an A/B test for the subject lines:

1. Go to Tools > Email marketing > A/B tests.
2. Click Create A/B test.
3. Select Subject line and click Create test button.
4. Provide your A/B test name – it will only be visible to you.
5. Select from and reply-to addresses.
6. Add your Subject lines variants. You can test up to 5 different subject lines to check which one gets you the most opens. For example, you can try personalizing your subject line, adding emojis, or including an offer.
   

   

7. Select your recipients by clicking Add recipients.
8. Design your message by clicking Design message. For more on using the new editor visit this section.
9. Click Configure A/B test to set up how you want the test to run. You can use the slider to decide on the size of the testing group. You can test it with up to 100% of your recipients. Using the percentage split, decide how large a group of contacts should receive the test variants and how large a group should receive the winning message. If you move the slider all the way to the right, 100% of contacts will receive test variants of your message, and the winning message will not be sent at all.
   
10. Select if you want us to send the winning message automatically to the rest of the group or if you’d rather do it manually (applicable only if less than 100% of contacts are selected for testing).
11. Click tracking will be enabled by default to see which contacts clicked your links. This will help you estimate engagement.
12. Select if you want to send clicks coming from the message as a source to Google Analytics.
13. Choose if you want to send the test immediately or at a scheduled time.
14. Now you can either save the message as a draft or start the A/B test.
    

    

How do I test the message content?

In order to create an A/B test for the content:

1. Go to Tools > Email marketing > A/B tests.
2. Click Create A/B test.
3. Select Content and click Create test button.
4. Provide your A/B test name – it will only be visible to you.
5. Select from and reply-to addresses and the subject line.
6. Select your recipients by clicking Add recipients.
7. Go to the Design and content section in order to create your content variants. Edit your content and test one thing at a time, for example, CTAs only. Use a template, start with a blank layout, or reuse your previous message. You need to create variant A first before proceeding to variant B. Click Create variant.

8. Once you create variant A, click Next, to move back to your message settings. Click Create variant next to message B, to create the another version of your message. Once you do that, click Next to get back to the message settings. You can create up to 5 variants for one A/B test.
9. Click Configure A/B test to set up how you want the test to run. You can use the slider to decide on the size of the testing group. You can test it with up to 100% of your recipients. Using the percentage split, decide how large a group of contacts should receive the test variants and how large a group should receive the winning message. If you move the slider all the way to the right, 100% of contacts will receive test variants of your message, and the winning message will not be sent at all.
   

10. Select if you want us to send the winning message automatically to the rest of the group or if you’d rather do it manually (applicable only if less than 100% of contacts are selected for testing). Next, decide what should be the winning factor (open rate or click rate) and after what time you want the test to end. We will show you a quick summary of your test group. Once you’re done, click Save.
    

11. Click tracking will be enabled by default to see which contacts clicked your links. This will help you estimate engagement.
12. Select if you want to send clicks coming from the message as a source to Google Analytics.
13. Choose if you want to send the test immediately or at a scheduled time.
14. Now you can either save the message as a draft or start the A/B test.

    

What statuses can my message have?

• Preparing – we are sending your message to the testing groups.
• Scheduled – the message is scheduled for delivery.
• Testing – we have sent the samples and are testing variants results.
• Testing finished – the samples have been tested and the results are ready to choose the winner (if you’ve chosen to manually select the winner).
• Sending winner – we are sending the winning message to the rest of the recipients.
• Cancelled – the message sending has been cancelled by the user.
• Completed – the test is completed and all the variants, as well as the winning message, have been sent and tested.
• Draft – the message is a draft and is waiting for completion.

How do I choose the winning message and where can I see it?

If you selected to choose the winner manually, you can expand your A/B test and then choose the winner from the Actions menu (vertical ellipsis) by clicking Send as winner.

You will be able to see your winning message above the variants. There you will see how many messages were delivered after the test stopped and what was the open rate and click rate for it. You will also be able to check detailed statistics for this message.

Where can I see the statistics?

You can see your message statistics if you go to Email marketing>> A/B tests. There you will be able to see the overview results for the A/B test (delivered messages, overall open rate, overall click rate). Once you expand the section, you’ll get the results divided into 2 categories:

1. Winner results.
2. Variant results.

Each of them will have delivered messages, open rate, and click rate statistics.
In the statistics for each variant, we will only show the opens and clicks generated during the duration of the test. For example, if the testing period was 2 days, we will only show data collected during these 2 days, not in the following days after the test ended.
Winner results will include delivered messages, open rate, and click rate both from the testing period and after sending the winner.

For more statistics on the winner, you can click View Statistics.

Legacy editor

How do I create an A/B test in the legacy editor?

Whenever you create a newsletter message in the legacy editor, you will find it at the bottom of the first page, in the Distribution settings section.

Turn on the A/B Test switch to enable it. Once enabled, the newsletter creator will be extended with additional steps where you define the type of test and its settings.

After you create the message in the editor, click Nest step. You will be redirected to the A/B test creation step where you can choose what you want to test in A/B test.

– For the subject, you can create up to 5 different subjects to test.
– For from field, you can test up to 4 different addresses.
– For content, you can choose or create up to 5 different messages.

After clicking Next step, you will be taken to the Choose recipients step where you can configure your test recipients and how the test and winner message will be delivered.

On the summary page, you will see which type of test you’ve chosen and a summary of your message configuration. Then you can send the test immediately or schedule it for later

Related articles:

Find more information about A/B tests on our Blog: A/B Testing Email Campaigns: All You Need to Know.