Introduction
- Have you ever wondered which subject line results in more opens?
- Would a red ‘Get Started’ button produce more clicks than a green one?
- Does having your company name in the sender name really make a difference?
Stop wondering and start getting answers with Mailjet’s A/B Testing!
What does A/B Testing mean exactly?
Through Mailjet’s A/B Testing (AB Testing), you are free to test almost anything within your email and compare up to 10 versions simultaneously.
Once you decide on what elements (such as subject line, sender name, images, buttons, …) you want to test and how many versions of the email you want to send, you are ready to launch the A/B Test.
You set the testing criteria (open rate, click rate, ...) and then each version is sent to a small but equal number of recipients from the same contact list.
Our powerful analytics tool shows real data in intuitive graphs, telling the story of which version resonated best with your customers.
You can then choose to send the winning version to the remaining recipients on that contact list, or have Mailjet do it for you automatically!
The bottom line with A/B Testing, you benefit from very efficient email testing and together with other features of Mailjet (Segmentation, Campaign Comparison), you will know how to design the most effective email and send it to the most rewarding group of recipients.
Creating Campaign Versions
Using the Mailjet’s A/B Testing feature is quite simple:
- Design your campaign
- Create one or more variations of that campaign
- Define the test criteria and recipient size
- Review the results
Let’s start by going to your Campaigns page and clicking the ‘New Campaign’ button.
Step 1: Setup
The 'Create a new campaign' page shows the different sections of your campaign that need to be completed:
- Campaign Name: This is an internal name for your campaign that only you see
- Language: Set the language of your campaign
- Contact List: Select your contact list. For premium plans users, create or apply a segmentation. For more information on segmentation, please read the FAQ: How to segment contact lists
-
Content: Design your content with our responsive Email Editor
- Subject: The subject of your campaign as it appears in your recipient's inbox
- From: Enter the name your recipients will see in the 'from' field, and the sending address
Click 'Design email' to select your campaign template and design your different versions with our Email builder.
Step 2: Design
Customize your campaign by adding new elements with a quick drag-and-drop. Click in any section to edit it. (For detailed information on how to use all the features, please read our Guide.)
Excellent! You have created your campaign, also known as Version A.
To create a new version, click on the drop down list, and select 'Create a new version'.
And Version B has been created.
Anytime a new version is created, it is an exact copy of the immediate previous. If you were to create Version C, it would be an exact copy of Version B, and so on.
To change the subject and sender details for Version B, click on the Edit icon.
To delete a version, simply click on the drop down list and select ‘Delete this version’.
(When you delete a version, any subsequent versions are renamed accordingly.)
Be careful on the number of changes you make in a single version. Having too many variables will make it difficult to determine which one variable made the most impact. Remember, you can create up to 10 versions of the same campaign.
Once you are ready to send your campaign, click on ‘Review & Send’.
Step 3: Test Sample & Winner Criteria
Now that you have created your campaign versions and back on the 'Create a new campaign' page, the next step is to set the test sample size and the criteria for selecting and sending the winning version.
The A/B Test Sample section will show the default configuration: 20% test sample size and the winning version is automatically send.
Click on the pencil icon to edit the configuration.
Test Sample Size
Enter the percentage of contacts that will receive your campaign versions. The total percentage or number of contacts will be divided by the number of versions you have.
You can now choose between two methods for your test sample:
-
Traditional A/B Testing: Test on a small sample, determine the winner, and then send the winning version to the rest of the list.
-
Split Send (New Feature): Directly send different versions to different segments of your list simultaneously.
Winner Selection (Excluding Split Send)
You can now select how the winning version will be sent: Automatically or Manually.
👉 With the Automatic mode, you select the criteria that will be used to determine the winning version at the end of the testing period (defined in the next step).
Once a winning version has been decided, Mailjet will automatically send it to the remainder of the contact list, and you will be notified via email.
In the event of a tie, Mailjet will choose the winning campaign with the best global score, and you will be notified via email.
👉 With the Manual mode, you will review the test results to select & send the winning version within 7 days of the initial sending.
Choose the desired mode, and then click ‘Save’.
Step 4: Schedule & Send
You can choose when to send your test sample by clicking ‘Schedule’.
If you selected the Automatic Mode, you can schedule when the test sample and winning version will be sent.
If you selected the Manual Mode, you can schedule when the test sample will be sent and set a reminder to analyze the results. Remember, the winning version needs to be manually sent within 7 days of the initial sending, otherwise the campaign will be canceled.
Click 'Send now' and your A/B Campaign is underway!
Accessing your A/B campaign
Regardless if your campaign is ongoing or completed, you can view it just like a regular campaign:
From your Campaigns page, simply click on the ‘A/B test’ tab then on your campaign title.
Viewing an ongoing A/B Test Auto Mode
With an ongoing A/B Test in Auto mode, you can:
- Change from Automatic to Manual mode, and then you can decide the winning version.
- Adjust the winning version criteria.
- Change the date or time when the winning version will be sent.
- Cancel the A/B Test altogether.
View an ongoing A/B Test Manual Mode
With an A/B in Manual mode, you can select the winning version at any time: From the drop-down list, select the winning version and then click ‘Send This Version’.
The other options with an ongoing A/B in Manual mode are:
- Change the date or time of your reminder email.
- Cancel the A/B Test completely.
View Completed A/B Test – Manual or Auto mode
Once the A/B Test has been completed, you will be able to view the powerful data to gain insight to what your customers liked and loved about your different versions.
The graph displays the selected data metric whereas the chart shows the details of all statuses for each version.
A) Change the display unit, time range, and chart type.
B) Select the desired metric to be displayed in the graph. And if you hover your mouse over the graph, a pop-up will show the KPIs for each campaign.
C) Toggle the campaign display on the graph by clicking on the name.
Kick up your analysis power to another level by comparing various campaigns and benchmarks. Please read the Campaign Comparison Guide for more information.