Experiment statistics collection and calculation

Understanding how the server collects and calculates statistics for the test elements in your marketing experiments can help you evaluate the results of the experiment.

The server displays the following statistics for the test elements in the experiment:

  • Unique customers (number of individual customers who reached the test element)
  • Views
  • View orders
  • View revenue
  • Clicks
  • Click orders
  • Click revenue

You can view these statistics while the experiment is running and after it ends. See Viewing the statistics of an experiment.

Enabling the collection of revenue statistics

To collect revenue statistics, you must enable the following statistical counters before your experiment starts:
  • ExperimentEvaluationEventListener
  • OrderSubmissionEvent
See Enabling the collection of marketing statistics in Management Center.

Frequency of statistical updates

The following table identifies how often statistics are updated in the Statistics tab while an experiment is running.

In the Views and Clicks columns The server gathers and displays these numbers using the SaveMarketingStatistics scheduled job. By default, this job runs every 30 minutes. You can change the frequency of this job; see Scheduling the SaveMarketingStatistics job.
In the View Orders, View Revenue, Click Orders and Click Revenue columns The server gathers and displays these numbers using the RaiseECEvent scheduled job. By default, this job runs every 5 minutes. You can change the frequency of this job; see Statistics considerations.

Difference between View and Click statistics

When interpreting statistical data about a web activity containing an experiment, it is important to understand the following terms:

  • View means the server displayed the results from an experiment test element to the customer in the e-Marketing Spot.
  • Click means the customer clicked the results displayed in the e-Marketing Spot.

In most cases, the click numbers are more meaningful because there is more certainty that the customer was interested in the test element if they clicked it. When interpreting the view numbers, you cannot assume that the customer actually saw the test element when the server displayed it. The view numbers, however, can give you some insight into how appealing the test element is. For example, you can calculate the click-through rate by dividing the number of times a user clicked a test element by the number of times the server displayed that element; the higher the percentage, the more successful the test element. Here is an example:

Test element Views Clicks Click-through rate
Advertisement A 500 100 20%
Advertisement B 1000 100 10%

These two advertisements were clicked the same number of times (100); however, the server displayed Advertisement B twice as many times as Advertisement A. The click-through rate indicates that Advertisement A is a more appealing advertisement.

How revenue numbers are calculated

As each customer successfully submits an order on your storefront, the server assesses the order to determine if there is any correlation between any items in the order and what the customer saw in any e-Marketing Spots used in any experiments in the current session. The server then calculates the revenue numbers using the logic described in the following table. In all cases:

  • The revenue numbers are updated based on the value of the entire order, not just the value of any products and categories associated with the results displayed in the e-Marketing Spot.
  • The customer's order must be successfully submitted within the timeframe the experiment specifies for the revenue numbers to be included in the statistics.
If Then
The experiment displays a catalog entry The revenue numbers are updated if the customer purchases the displayed catalog entry.
The experiment displays a category The revenue numbers are updated if the customer purchases any item from the displayed category.
The experiment displays an advertisement
  • If the advertisement is associated with a catalog entry (the click action is Display catalog entry), revenue numbers are updated if the customer purchases the displayed catalog entry.
  • If the advertisement is associated with a category (the click action is Display category), the revenue numbers are updated if the customer purchases any item from the displayed category.
  • If the advertisement is not associated with any catalog entry or category, revenue numbers are updated if a customer purchases anything during the session.

Experiment statistics in a staging environment

Marketing statistics are captured on the production server. If you want to view experiment statistics on a staging server for analysis, a System Administrator can transfer the statistical data from the production server to the staging server by running the ExportStats and ImportStats commands. When running the commands, specify the DMEXPSTATS and DMELESTATS tables, which contain e-Marketing Spot statistics. See Copying marketing statistics from the production server.