Manage A/B Testing

Add A/B testing to treatment rules

To add or edit an A/B testing to a treatment rule, complete the following steps:

1. On right pane of Strategy tab, click Add/Modify Rules which takes you to Edit mode.
2. Select a treatment rule in the list for which you want to define A/B Testing, hover the mouse over the three dots at most right and click A/B. Advanced Options panel appears with A/B testing tab is landed.
3. Select the Enabled checkbox.

   This adds base rule as first branch in Branch table. You can add more branches and other settings once A/B testing is enabled. If you are not ready to run the A/B Testing yet, you can disable it and your settings will still be saved. However, only enabled A/B Testings are deployed during strategy deployment and executed in Runtime.

4. Provide appropriate values for Effective Date and Expiration Date.

   An A/B testing is executed in Runtime in the period of time defined by effective date and expiration date. Both dates are optional. If the effective date is not defined, then A/B testing is in effect immediately after deployment. If expiration date is not defined, then it is in effect forever.

5. For Selection Mode, select either:
   • Use default: Select this option if you want the offer to be served from the branch you defined out of A/B test branches, regardless of acceptance rate of any offer.
   • Auto selection: Select this option if you want the offer to be served from one of the A/B test branches having highest acceptance rate. Acceptance rate is calculated as ratio of accepted count to contact count.
   • Use base: Select this option if you want the offer to be served from original treatment rule, regardless of acceptance rate of any offer.
6. Add more branches by click + Add Branch. An A/B testing must have at lease two branches. You can also delete unwanted branches by hovering the mouse over the three dots, at the end of the branch, and selecting the trash can icon.
7. Branch Name is an unique identifier for a branch. It is a required field and must be unique within A/B testing branches. You can use generated name or specify your own.
8. Cell Code, used for A/B testing report, is unique for a branch. You can use generated cell code or specify your own.
9. Adjust the sampling percentage so that total percentage of all active branches adds up to a hundred. For example, you have two branches you can assign 50 to each branch. If you have three branches you can assign 40, 30, and 30 to the branches.
10. Select an offer to serve for each branch.
11. Optionally, specify Offer Attributes values for each branch in Edit Offer Attribute panel. This could be used where you want to test same offer, but only an offer attribute, saying sale price, being different in branches.
12. Click Save on Advanced Options panel to exit it.
13. Click Save on Treatment Rule Edit pane and save A/B Testing definition of the Treatment Rule to database.
14. Deploy the changes. For Deployment, Deploy interaction strategy and its sub-topics.

View A/B Testing

In Treatment Rule list of Strategy tab, for any rule that has A/B testing defined, there is an A/B Testing icon located on the right. An orange color means the A/B testing is active and gray color means it is disabled.

On top of the list, there is a checkbox Show A/B testing rules only. Clicking the checkbox will take you to a view where only rules having A/B Testing defined are displayed. You can click Up and Down arrow to expand and collapse and see branches for an A/B Testing rule. There are also two columns, Selection Mode and Acceptance Rate, are displayed in this view.

Fetch Acceptance Rate for A/B Testing

In Interact Runtime, when a contact or an accepted event is posted for a treatment and if this treatment is a branch from an A/B Testing, Interact increments contactCount or acceptCount for that branch in UACI_ABTestPerformance table of Runtime database. Acceptance Rate of a branch is ratio of acceptCount to contactCount of the branch. In A/B Testing view (checking Show A/B testing rules only checkbox), Acceptance Rates of an A/B testing are fetched only on demand by clicking Get acceptance rate of any branch of the A/B Testing.

View A/B Testing Report

To view the A/B Testing report, complete the following steps:

1. Navigate to Interact > Interactive channels. The All interactive channels page opens.
2. Select the required interactive channel from the list. The page related to the interactive channel opens.
3. Select the Analysis tab.
4. In the Report type dropdown box, select A/B Test Performance Analysis.
5. The report appears on screen. If required, you can export the report to PDF or XSLX format.
6. Select Export report. A Export Report panel appears. Perform the following steps:
   1. Select the Export Format. Available options are PDF or XLSX.
   2. For Pages section, select either:
      • All pages: Exports all pages.
      • Current page: Exports only the current page.
      • Pages: Exports the page range mentioned.
   3. For Fit To section, select either:
      • Auto: Automatically fits the page size that is optimal for export.
      • Actual size: Exports actual page size.
      • Fit to whole page: Exports the size that fits the whole page.
   4. Click OK.