Splitting rules

Launch A/B experiments with the clear picture of what is going on under the hood.

There is often a need to run a few experiments simultaneously:

  • There is enough traffic to validate a few hypotheses at a time.
  • There are separate teams responsible for onboarding experience, paywalls, and monetisation strategy.

Knowing the details of how Qonversion Experiments distributes users between tests is crucial for your work. Let's consider the main rules and a couple of examples demonstrating how users are assigned into variants in Qonversion Experiments.

General splitting rules

Splitter in Qonversion Experiments works in the following way:

  1. At the top of the processing funnel, Splitter receives User_A.
  2. Splitter sorts all of the launched experiments by start date and checks if User_A can be assigned to each experiment.
    1. Please note that a User can be assigned only to one experiment at a time.
  3. When Splitter validates if User_A matches experiment rules, it checks the following attributes:
    1. Segmentation rules (Country, App version, etc.)
    2. Level of traffic allocation (30%, 50%, 70%, etc.)
  4. If User_A matches segmentation rules, he is assigned to a specific group inside the experiment. Otherwise, Splitter returns to Step #2 to check User_A against the following experiment in the queue.
    1. Please note, if you have the traffic level in the first launched experiment (sorted by start date) set to 70% and in the second - 30%, that means that your second experiment is exposed for (100% - 70%) * 30% = 9% of all users. Set the traffic level to 100% for the second experiment in case you want to use all the remaining from the first experiment users. Below you can find a couple of examples that help understand this approach better.

Example #1

Let's consider the case of launching two experiments at a time:

  1. One for users from the UK, 100% traffic allocation level. Start date = 09.03.23
  1. One for users from the US. 100% traffic allocation level. Start date = 15.03.23.

As a result

  • 100% of users from the UK are assigned to experiment #1.
  • 100% of users from the US are assigned to experiment #2.
  • All the remaining users are not assigned to any experiment.
  • Sorting based on the Start date does not influence splitting results (step#2 in General splitting rules).

Example #2

Let's consider the case of launching two experiments at a time but with similar segmentation settings:

  1. One for users from the UK, 70% traffic allocation level. Start date = 09.03.23
  1. Another one is for users from the UK as well. 30% traffic allocation level. Start date = 15.03.23.

Based on the General splitting rules, you have the following distribution picture:

  • 70% of users from the UK are assigned to Experiment #1 (with Start date = 09.03.23).
  • (100% - 70%) * 30% = 9% of users from the UK are assigned to Experiment #2 (with Start date = 15.03.23).
  • 100% - 70% - 9% = 21% of users from the UK are not assigned to any experiment.
  • 100% of users outside the UK are not assigned to any experiment.