Email Send: A/B Testing (Split Testing)

Set Up an A/B Test

In this example, each version will be sent to 500 recipients. At the end of the 5 hours the system will choose a winner based on either the most unique opens or unique clicks (your choice).  The winning version, A or B, will then be sent to the remaining 44,000 recipients.  The benefit is that the vast majority of your recipients will receive the better performing email version and you will learn what caused the increased engagement.

To set up an A/B Test, you need to have already created your Email Send and added at least 3 recipients. On the Email Send record, choose A/B Test.

The A/B Testing Wizard will open. On the left are the details for the Email Send you've created, which is automatically assigned as Strategy A. The information in Strategy A cannot be edited. On the right side is the information for Strategy B. At least one of the fields should be edited (Subject, Preheader, From Name, From Address, or From Record Owner).

You can also choose a different template as well by using the lookup field to select an existing template. If you would like to edit the email content for Strategy B, click on the Edit button. This will open the editor an allow you to make changes just for version B. Just be sure to save!

If using the Send Time variable in your test, you can select a scheduled date/time on the Email record for version A to send at, and then in the A/B Test window select your version B send time. For version B, just switch Send Immediately to No and you'll see the date/time field display so you can designate the scheduled send time.

The timezone being used in the Send Time selector reflects the current user's timezone.

Click Next when you're ready to move to the next page of the A/B Test Wizard.

This page allows you to set the size of the test groups and winner group. The test and winning groups are split 50/50 by default: 25% to each test group and the other 50% to the winning group. To edit the size of the test groups and winning group, slide the black slider bar to the left for smaller test groups...

or right for larger test groups.

Winning Strategy indicates how the winning test version will be determined. This can be set as the version which receives the most unique clicks, or the version that which receives the most unique opens.

NOTE: You must have a winning strategy for the A/B Test to function properly. If your Winning Strategy is blank or if your Winning Version is 0% of recipients, the A/B Test will not work correctly.

Select Winning Strategy Automatically denotes whether the system should automatically select and send the winning version, or if you would like to manually select and send the winning version. If this field is set to Yes, you can set a value in Select a Winner After to indicate how long the system should wait before sending the winning version to the winning version group of recipients.

If Select Winning Strategy Automatically is set to No, you can wait for however long you would like for the A/B Test to run, then when you are ready to select a winning version, return to the Email Send record again, click Send again, and you will be shown a screen with the results for version A and version B that prompts you to select a winning version to send.

Winning Strategy for Send Time testing

If using the Send Time in your testing, the Winning Strategy is different than all the other testing options. On the second screen of the A/B Test window, you'll select your test group size as usual, but then you'll only select the Winning Strategy criteria—either Unique Click Rate or Unique Open Rate. The option to select a winning strategy automatically will not be accessible; you will see it but you won't be able to set it to Yes or change the time for a winner to be selected. 

Instead, you will need to manually click the Send button on your Email, once you are ready to send the winning version. The approval screen will tell you which strategy was the winner, but you will be able to choose either strategy as your final send to your remaining recipients. 

Once the fields are populated, choose Save and Send to begin the A/B Test or chose Save and go back to the Email Send record.

Viewing A/B Test Data

The information entered in the A/B Test Wizard can also be found under A/B Test Preferences in the Email Send record.

NOTE: The A/B Test Preferences on the Email Send record will show the details of how the A/B Test was set up in the Wizard. The fields are not to be manually filled as it will not trigger an A/B Test.

To check on the progress of the A/B Test, go to Messaging  > Sent, change the view to A/B Test, open the A/B Test record and scroll down to Email Statistics. The A/B Test section of the Email Statistics shows Unique Clicks and Unique Opens for both Strategy A and Strategy B. Read more about A/B Test Statistics.

(Dynamics 2013/2015) Set Up an A/B Test

In this example, each version will be sent to 500 recipients. At the end of the 5 hours the system will choose a winner based on either the most unique opens or unique clicks (your choice).  The winning version, A or B, will then be sent to the remaining 44,000 recipients.  The benefit is that the vast majority of your recipients will receive the better performing email version and you will learn what caused the increased engagement.

To set up an A/B Test, you need to have already created your Email Send and added at least 3 recipients. On the Email Send record, choose A/B Test.

The A/B Testing Wizard will open. On the left are the details for the Email Send you've created, which is automatically assigned as Strategy A. The information in Strategy A cannot be edited. On the right side is the information for Strategy B. At least one of the fields should be edited (Subject, Preheader, From Name, From Address, or From Record Owner).

You can also choose a different template as well by using the lookup field to select an existing template. If you would like to edit the email content for Strategy B, click on the Edit button. This will open the editor an allow you to make changes just for version B. Just be sure to save!

If using the Send Time variable in your test, you can select a scheduled date/time on the Email record for version A to send at, and then in the A/B Test window select your version B send time. For version B, just switch Send Immediately to No and you'll see the date/time field display so you can designate the scheduled send time.

The timezone being used in the Send Time selector reflects the current user's timezone.

Click Next when you're ready to move to the next page of the A/B Test Wizard.

This page allows you to set the size of the test groups and winner group. The test and winning groups are split 50/50 by default: 25% to each test group and the other 50% to the winning group. To edit the size of the test groups and winning group, slide the black slider bar to the left for smaller test groups...

or right for larger test groups.

Winning Strategy indicates how the winning test version will be determined. This can be set as the version which receives the most unique clicks, or the version that which receives the most unique opens.

Select Winning Strategy Automatically denotes whether the system should automatically select and send the winning version, or if you would like to manually select and send the winning version. If this field is set to Yes, you can set a value in Select a Winner After to indicate how long the system should wait before sending the winning version to the winning version group of recipients.

If Select Winning Strategy Automatically is set to No, you can wait for however long you would like for the A/B Test to run, then when you are ready to select a winning version, return to the Email Send record again, click Send again, and you will be shown a screen with the results for version A and version B that prompts you to select a winning version to send.

Winning Strategy for Send Time testing

If using the Send Time in your testing, the Winning Strategy is different than all the other testing options. On the second screen of the A/B Test window, you'll select your test group size as usual, but then you'll only select the Winning Strategy criteria—either Unique Click Rate or Unique Open Rate. The option to select a winning strategy automatically will not be accessible; you will see it but you won't be able to set it to Yes or change the time for a winner to be selected. 

Instead, you will need to manually click the Send button on your Email, once you are ready to send the winning version. The approval screen will tell you which strategy was the winner, but you will be able to choose either strategy as your final send to your remaining recipients. 

Once the fields are populated, choose Save and Send to begin the A/B Test or chose Save and go back to the Email Send record.

Viewing A/B Test Data

The information entered in the A/B Test Wizard can also be found under A/B Test Preferences in the Email Send record.

NOTE: The A/B Test Preferences on the Email Send record will show the details of how the A/B Test was set up in the Wizard. The fields are not to be manually filled as it will not trigger an A/B Test.

To check on the progress of the A/B Test, go to ClickDimensions > Emails, change the view to A/B Test, open the A/B Test record and scroll down to Email Statistics. The A/B Test section of the Email Statistics shows Unique Clicks and Unique Opens for both Strategy A and Strategy B. Read more about A/B Test Statistics.