A/B testing of treatment rules

A/B testing, also known as 'split testing' is the process of comparing two versions of same scenario with just one varying parameter. It brings scientific methodology to marketing and removes the guesswork.

In Interact, A/B testing is the test done for a treatment rule with all properties fixed, except one varying parameter. In version 12.1 and higher releases, this parameter is the combination of offer and/or offer attributes. In an A/B testing of a rule, each variation of offer and/or offer attributes represents one A/B testing branch within same audience distribution and same zone. One treatment rule can only have one A/B Testing defined. One A/B testing can have multiple branches defined depending on the marketers testing requirements. But un-necessary branches can degrade performance in the runtime process.

An A/B testing is executed in Runtime in the period of time defined by effective date and expiration date. Both dates are optional. If the effective date is not defined, then A/B testing is in effect immediately after deployment. If expiration date is not defined, then it is in effect forever.

In the period of A/B testing, one of the offers from A/B testing branches is randomly served to visitors. Once A/B testing is completed, i.e., after its expiration date, the branch which remains in execution is determined by the Selection mode specified by marketers in A/B testing definition. The following three selection modes are supported.

• Default branch: User selects a branch from all branches in an A/B testing definition.
• Auto selection: It is based on the history of acceptance data. Interact runtime automatically selects the branch that has the highest acceptance rate.
• Use base: Base treatment rule is used so that arbitration logic is resumed as it was before the A/B testing.

In Interact runtime, all branches in an A/B testing are treated like treatment rules. When a contact or an accepted event is posted for an offer and the offer comes from one of branches of an A/B test, Interact runtime update contact count or accepted count for the branch in UACI_ABTestPerformance table in Runtime database. The acceptance rate of a branch is defined as ratio of number of branch offer being accepted to number of the offer being contacted. At the end of A/B testing, counts in UACI_ABTestPerformance table are used to calculate the acceptance rates for all branches of the A/B testing. The ultimate goal of an A/B testing is to find a branch or a treatment rule (combination of an offer and offer attributes), which has the best acceptance rate from visitors.