A/B Split and Nth-select Testing

From Enterprise Help
This is the approved revision of this page, as well as being the most recent.
Jump to: navigation, search

Enterprise provides an extremely flexible interface for performing Nth Select and A/B split testing. Nth Select testing allows you to identify a small test panel or several small test panels within your list. You can then deploy distinct mailings utilizing different creative, subject lines and other message elements to your test panels. Based on response data in Reports, you can identify which test panel had the highest response rate, and send this message to the remainder of your list. A/B split testing allows you to split your list into segments of equal size and then send different messages to each "split" -- this is a slightly different testing methodology than Nth select because all recipients of the list will receive a version of the message at the same time, and effectiveness of the various messages will be analyzed after the entire campaign has run. Below are examples for running both A/B split and Nth Select tests using Enterprise segments.

A/B Split Testing

To conduct a classic A/B Split test, we first segment our Members into two groups of equal size. We achieve this by clicking the "A/B" button in the toolbar on the segment screen. We'll create our "A" group first by defining a segment which specifies that everyone is this Segment is "in Group 1 of 2." This means that Enterprise will divide our Members into two groups of equal or nearly equal size, and that this segment will contain all Members in the first of these two groups.


Once we've created this segment, we click "Save", and then add another segment. This time, we specify that the segment should contain all Members "in Group 2 of 2".


We then save this segment, and we now have two groups of Members each of which contains half of the Site's Members. We can now create two mailings with different subject lines, offers, creative and so forth to test the effectiveness of various elements. We'll send one of the mailings to our "Split_A" segment and the other to our "Split_B" Segment. We can then compare the results of the mailings in Reports to see which mailing was more effective.

Enterprise allows you to go beyond simple A/B testing by allowing you to create up to 100 different splits. This would allow you, for example, to create five different segments (1 of 5, 2 of 5, 3 of 5, etc.) to test 5 different offers.

Nth Select Testing

Another standard form of testing is Nth Select testing, in which you test creative or offers to one or more small test panels and then send the best performing message to the remainder of your list. To conduct an Nth Select test, you begin by defining your test panels. In the example below, we will create two test panels.

Let's say that we have 100,000 Members in our Site, and that we want to send two different versions of a message to two different test panels of 1000 Members each.We begin by defining our two test panels. The first test panel will contain all Members "in Group 1 of 100."


Once we've created this segment we click "Save" and then create another segment containing all Members "in Group 2 of 100."


We now have two test panels which contain 1,000 Members each. We have divided our Members into 100 groups and defined two test panels, each of which contains the Members for one of those groups. Since each group contains 1 out of every 100 Members, and we have 100,000 Members, our test panels will contain 1,000 Members each.

The next step is to create two different mailings, and send one mailing to Test Panel 1 and the other mailing to Test Panel 2. Once we've analyzed the results in Reports, we can send the best-performing message to the remainder of our Members. To do this, we first create a segment which contains all Members who were not part of Test Panel 1 or 2. To do this we create a Segment which contains all Members "In any Group greater than 2 of 100."


Once we have created the segment, we can create a new mailing using the desired content and send it to this segment.