DocsAPI ReferenceDiscussionsBlogGithub
Login

Launch experiments

Plan, configure, and launch monetisation experiments with flexible A/B testing solution

Suggest Edits

This guide helps you walk through five main steps to launch your experiment:

  1. Plan your experiment
  2. Configure and test variants
  3. Segment users
  4. Adjust traffic level
  5. Launch the experiment

Plan your experiment

Writing down your hypothesis before configuring and starting the experiment is essential. This helps to avoid several biases (for instance, confirmation bias) during experiment analyses.

Choose your Primary Metric

You can select the Primary Metric for the experiment from the metrics that Qonversion analytics tracks:

  • User-to-Trial Conversion
  • User-to-Paid Conversion
  • New Trials
  • Trial-to-Paid Conversion
  • Trials Cancellation Rate
  • New Subscriptions
  • Subscriptions Cancellation Rate
  • Sales
  • Proceeds
  • Refunds

Configure and test variants

Qonversion Experiments provides you with flexible in-app values using which you can set up any behavior and appearance of your app. The core of these in-app values is a plain JSON file consisting of key-value data. We support the following data types:

  • String.
    • Use this option to validate pricing, communication, or visual hypothesis.
    • Examples: native.subs.full.v4.w.8.99.trial.7d, Unlock Fast and Secure Browsing, #3076FF, etc.
  • Number.
    • Use this option, for example, to show only a subset of your onboarding screens.
    • Examples: 2, 5, 23, etc.
  • Boolean.
    • Use this option to turn on or off some features.
    • Examples: true, false
  • Json.
    • Use this option to validate advanced changes in your app's behavior.
    • Examples: {"banner_text": "Choose your plan", "product_top": "native.subs.full.v4.w.8.99.trial.7d", "product_bottom": "subs.month.17.99", "skip_onboarding": true}.

Integrate changes into your app

  1. Prerequisites, skip this step if you have already integrated Qonversion SDK. To enable Qonversion to run and calculate A/B testing results, you need to have Qonversion SDK installed in your app.

  2. Call the Qonversion remoteConfig method.

    Qonversion.shared().remoteConfig { remoteConfig, error in
    // JSON payload you have configured using the Qonversion dashboard.
    remoteConfig.payload
                                  
    // Source of the configuration. 
    remoteConfig.source

    // One of the following: experiment control group, experiment treatment group, remote configuration.
    remoteConfig.source.type
    
    // Either experiment's or remote config's ID.
    remoteConfig.source.identifier
                            
    // Either experiment's or remote config's name.
    remoteConfig.source.name              
	                
    // How the current payload was assigned to the user. Either automatically or manually.
    remoteConfig.source.assignmentType         
                 
    // Object with the experiment's information.                         
    remoteConfig.experiment
                                  
    // Experiment's identifier.                              
    remoteConfig.experiment.identifier 
                                  
    // Experiment's name. The same as you set in Qonversion. You can use it for analytical purposes.
    remoteConfig.experiment.name
                                  
    // Experiment's group the user has been assigned to.
    remoteConfig.experiment.group
                                  
    // Experiment group's identifier.
    remoteConfig.experiment.group.identifier
                                  
    // Experiment group's name. The same as you set in Qonversion. You can use it for analytical purposes.                              
    remoteConfig.experiment.group.name
                                  
    // Type of the experiment's group. Either control or treatment.                              
    remoteConfig.experiment.group.type                         
}

    [[Qonversion sharedInstance] remoteConfig:^(QONRemoteConfig * _Nullable remoteConfig, NSError * _Nullable error) {
    // JSON payload you have configured using the Qonversion dashboard.
    remoteConfig.payload
           
    // Source of the configuration. 
    remoteConfig.source

    // One of the following: experiment control group, experiment treatment group, remote configuration.
    remoteConfig.source.type
    
    // Either experiment's or remote config's ID.
    remoteConfig.source.identifier
                            
    // Either experiment's or remote config's name.
    remoteConfig.source.name              
	                
    // How the current payload was assigned to the user. Either automatically or manually.
    remoteConfig.source.assignmentType         
      
    // Object with the experiment's information.                            
    remoteConfig.experiment
                                  
    // Experiment's identifier.                              
    remoteConfig.experiment.identifier 
                                  
    // Experiment's name. The same as you set in Qonversion. You can use it for analytical purposes.
    remoteConfig.experiment.name
                                  
    // Experiment's group the user has been assigned to.
    remoteConfig.experiment.group
                                  
    // Experiment group's identifier.
    remoteConfig.experiment.group.identifier
                                  
    // Experiment group's name. The same as you set in Qonversion. You can use it for analytical purposes.                              
    remoteConfig.experiment.group.name
                                  
    // Type of the experiment's group. Either control or treatment.                              
    remoteConfig.experiment.group.type                               
}];

    Qonversion.shared.remoteConfig(object : QonversionRemoteConfigCallback {
    override fun onSuccess(remoteConfig: QRemoteConfig) {
        //JSON payload you have configured using the Qonversion dashboard.
        remoteConfig.payload
      
        // Source of the configuration. 
        remoteConfig.source

        // One of the following: experiment control group, experiment treatment group, remote configuration.
        remoteConfig.source.type

        // Either experiment's or remote config's ID.
        remoteConfig.source.identifier

        // Either experiment's or remote config's name.
        remoteConfig.source.name              

        // How the current payload was assigned to the user. Either automatically or manually.
        remoteConfig.source.assignmentType         
                 
        // Object with the experiment's information.                            
        remoteConfig.experiment
                                  
        // Experiment's identifier.                              
        remoteConfig.experiment.id 
                                  
        // Experiment's name. The same as you set in Qonversion. You can use it for analytical purposes.
        remoteConfig.experiment.name
                                  
        // Experiment's group the user has been assigned to.
        remoteConfig.experiment.group
                                  
        // Experiment group's identifier.
        remoteConfig.experiment.group.id
                                  
        // Experiment group's name. The same as you set in Qonversion. You can use it for analytical purposes.                              
        remoteConfig.experiment.group.name
                                  
        // Type of the experiment's group. Either control or treatment.                              
        remoteConfig.experiment.group.type                              
    }

    override fun onError(error: QonversionError) {
        // handle error
    }
})

    Qonversion.getSharedInstance().remoteConfig(new QonversionRemoteConfigCallback() {
    @Override
    public void onSuccess(@NonNull QRemoteConfig remoteConfig) {
        //JSON payload you have configured using the Qonversion dashboard.
        remoteConfig.getPayload()
          
        // Source of the configuration.
        remoteConfig.getSource()
          
        // One of the following: experiment control group, experiment treatment group, remote configuration.
        remoteConfig.getSource().getType()
    
        // Either experiment's or remote config's ID.
        remoteConfig.getSource().getIdentifier()
                            
        // Either experiment's or remote config's name.
        remoteConfig.getSource().getName()
                                  
        // How the current payload was assigned to the user. Either automatically or manually.
        remoteConfig.getSource().getAssignmentType()                                  
                 
        // Object with the experiment's information.                            
        remoteConfig.getExperiment()
                                  
        // Experiment's identifier.                              
        remoteConfig.getExperiment().getId()
                                  
        // Experiment's name. The same as you set in Qonversion. You can use it for analytical purposes.
        remoteConfig.getExperiment().getName()
                                  
        // Experiment's group the user has been assigned to.
        remoteConfig.getExperiment().getGroup()
                                  
        // Experiment group's identifier.
        remoteConfig.getExperiment().getGroup().getId()
                                  
        // Experiment group's name. The same as you set in Qonversion. You can use it for analytical purposes.                              
        remoteConfig.getExperiment().getGroup().getName()
                                  
        // Type of the experiment's group. Either control or treatment.                              
        remoteConfig.getExperiment().getGroup().getType()
    }

    @Override
    public void onError(@NonNull QonversionError error) {
        // handle error
    }
});

    const remoteConfig = await Qonversion.getSharedInstance().remoteConfig();

//JSON payload you have configured using the Qonversion dashboard.
remoteConfig.payload

// Source of the configuration. 
remoteConfig.source

// One of the following: experiment control group, experiment treatment group, remote configuration.
remoteConfig.source.type

// Either experiment's or remote config's ID.
remoteConfig.source.identifier

// Either experiment's or remote config's name.
remoteConfig.source.name              

// How the current payload was assigned to the user. Either automatically or manually.
remoteConfig.source.assignmentType         
                 
// Object with the experiment's information.                            
remoteConfig.experiment
                                  
// Experiment's identifier.                              
remoteConfig.experiment.id 
                                  
// Experiment's name. The same as you set in Qonversion. You can use it for analytical purposes.
remoteConfig.experiment.name
                                  
// Experiment's group the user has been assigned to.
remoteConfig.experiment.group
                                  
// Experiment group's identifier.
remoteConfig.experiment.group.id
                                  
// Experiment group's name. The same as you set in Qonversion. You can use it for analytical purposes.                              
remoteConfig.experiment.group.name
                                  
// Type of the experiment's group. Either control or treatment.                              
remoteConfig.experiment.group.type                             

    final remoteConfig = await Qonversion.getSharedInstance().remoteConfig();

//JSON payload you have configured using the Qonversion dashboard.
remoteConfig.payload

// Source of the configuration. 
remoteConfig.source

// One of the following: experiment control group, experiment treatment group, remote configuration.
remoteConfig.source.type

// Either experiment's or remote config's ID.
remoteConfig.source.identifier

// Either experiment's or remote config's name.
remoteConfig.source.name              

// How the current payload was assigned to the user. Either automatically or manually.
remoteConfig.source.assignmentType         
                 
// Object with the experiment's information.                            
remoteConfig.experiment
                                  
// Experiment's identifier.                              
remoteConfig.experiment.id 
                                  
// Experiment's name. The same as you set in Qonversion. You can use it for analytical purposes.
remoteConfig.experiment.name
                                  
// Experiment's group the user has been assigned to.
remoteConfig.experiment.group
                                  
// Experiment group's identifier.
remoteConfig.experiment.group.id
                                  
// Experiment group's name. The same as you set in Qonversion. You can use it for analytical purposes.                              
remoteConfig.experiment.group.name
                                  
// Type of the experiment's group. Either control or treatment.                              
remoteConfig.experiment.group.type                                 

    Qonversion.GetSharedInstance().RemoteConfig((remoteConfig, error) =>
{
    //JSON payload you have configured using the Qonversion dashboard.
    remoteConfig.Payload
                 
    // Source of the configuration.
    remoteConfig.Source
    
    // One of the following: experiment control group, experiment treatment group, remote configuration.
    remoteConfig.Source.Type
                            
    // Either experiment's or remote config's ID.
    remoteConfig.Source.Identifier
      
    // Either experiment's or remote config's name.
    remoteConfig.Source.Name
                                  
    // How the current payload was assigned to the user. Either automatically or manually.
    remoteConfig.Source.AssignmentType
      
    // Object with the experiment's information.                            
    remoteConfig.Experiment
                                  
    // Experiment's identifier.                              
    remoteConfig.Experiment.Id 
                                  
    // Experiment's name. The same as you set in Qonversion. You can use it for analytical purposes.
    remoteConfig.Experiment.Name
                                  
    // Experiment's group the user has been assigned to.
    remoteConfig.Experiment.Group
                                  
    // Experiment group's identifier.
    remoteConfig.Experiment.Group.Id
                                  
    // Experiment group's name. The same as you set in Qonversion. You can use it for analytical purposes.                              
    remoteConfig.Experiment.Group.Name
                                  
    // Type of the experiment's group. Either control or treatment.                              
    remoteConfig.Experiment.Group.Type                                
});

    const remoteConfig = await Qonversion.getSharedInstance().remoteConfig();

//JSON payload you have configured using the Qonversion dashboard.
remoteConfig.payload

// Source of the configuration. 
remoteConfig.source

// One of the following: experiment control group, experiment treatment group, remote configuration.
remoteConfig.source.type

// Either experiment's or remote config's ID.
remoteConfig.source.identifier

// Either experiment's or remote config's name.
remoteConfig.source.name              

// How the current payload was assigned to the user. Either automatically or manually.
remoteConfig.source.assignmentType
                 
// Object with the experiment's information.                            
remoteConfig.experiment
                                  
// Experiment's identifier.                              
remoteConfig.experiment.id 
                                  
// Experiment's name. The same as you set in Qonversion. You can use it for analytical purposes.
remoteConfig.experiment.name
                                  
// Experiment's group the user has been assigned to.
remoteConfig.experiment.group
                                  
// Experiment group's identifier.
remoteConfig.experiment.group.id
                                  
// Experiment group's name. The same as you set in Qonversion. You can use it for analytical purposes.                              
remoteConfig.experiment.group.name
                                  
// Type of the experiment's group. Either control or treatment.                              
remoteConfig.experiment.group.type
  • Please note, that Qonversion assigns eligible users to launched experiments once the remoteConfig is called. We recommend paying attention to the following:
    • The point in the user flow where the method is called. The closer it is to the experiment behavior, the more precise the results will be.
    • Segmentation rules for the experiment.
  1. Adjust your app behavior and appearance based on the previously set in-app values.

Test changes before launch

We highly recommend testing if your app works properly with the changes you want to roll out to your users. You can do so by following the following steps:

  1. Copy the ID for your experiment and the variant you want to test.
  1. Pass the values to Qonversion SDK by using the attachUserToExperiment method
Qonversion.shared().attachUser(toExperiment: "your_experiment_id", groupId: "your_group_id") { success, error in
     // handle result
}

[[Qonversion sharedInstance] attachUserToExperiment:@"your_experiment_id" groupId:@"your_group_id" completion:^(BOOL success, NSError * _Nullable error) {
    // handle result
}];

Qonversion.shared.attachUserToExperiment("your_experiment_id", "your_group_id", object : QonversionExperimentAttachCallback {
    override fun onSuccess() {
        // handle success
    }

    override fun onError(error: QonversionError) {
        // handle error
    }
})

Qonversion.getSharedInstance().attachUserToExperiment("your_experiment_id", "your_group_id", new QonversionExperimentAttachCallback() {
    @Override
    public void onSuccess() {
        // handle success
    }

    @Override
    public void onError(@NonNull QonversionError error) {
        // handle error
    }
});

await Qonversion.getSharedInstance().attachUserToExperiment("your_experiment_id", "your_group_id");

await Qonversion.getSharedInstance().attachUserToExperiment("your_experiment_id", "your_group_id");

Qonversion.GetSharedInstance().AttachUserToExperiment("your_experiment_id", "your_group_id", (success, error) =>
{
    // handle result
});

await Qonversion.getSharedInstance().attachUserToExperiment("your_experiment_id", "your_group_id");
  • In case your user has already been attached to another experiment, use the detachUserFromExperiment method.
Qonversion.shared().detachUser(fromExperiment: "attached_experiment_id") { success, error in  
    // handle result  
}

[[Qonversion sharedInstance] detachUserFromExperiment:@"attached_experiment_id" completion:^(BOOL success, NSError * _Nullable error) {
    // handle result
}];

Qonversion.shared.detachUserFromExperiment("attached_experiment_id", object : QonversionExperimentAttachCallback {
    override fun onSuccess() {
        // handle success
    }

    override fun onError(error: QonversionError) {
        // handle error
    }
})

Qonversion.getSharedInstance().detachUserFromExperiment("attached_experiment_id", new QonversionExperimentAttachCallback() {
    @Override
    public void onSuccess() {
        // handle success
    }

    @Override
    public void onError(@NonNull QonversionError error) {
        // handle error
    }
});

await Qonversion.getSharedInstance().detachUserFromExperiment("attached_experiment_id");

await Qonversion.getSharedInstance().detachUserFromExperiment("attached_experiment_id");

Qonversion.GetSharedInstance().DetachUserFromExperiment("attached_experiment_id", (success, error) =>
{
    // handle result
});

await Qonversion.getSharedInstance().detachUserFromExperiment("attached_experiment_id");
  1. Call the Qonversion remoteConfig method. Now Qonversion SDK returns data associated with previously set experiment and variant.
  2. Validate your app logic without starting the experiment.

📘

Pay attention before release

Do not forget to remove the usage of the attachUserToExperiment method before your app release. Otherwise, all your users may be exposed to only one experiment group.

Segment users

Qonversion Experiments tool has a flexible set of options to run experiments to a desired user segment.

Segment users by:

  • App install date
    • Use this option to run the experiment only on new app installs.
  • App version
    • Use this option to run experiments only for users with the app version with the new features implemented.
    • For example, you want to test how the new paywall influences your revenue, and only app version 5.3.1 and above supports it. Thus, having users with lower app versions in the experiment only adds irrelevant data noise.
  • Country
    • Use this option to validate your app pricing or new behavior only in selected countries.
  • Store
    • Use this option to target only a specific store (Apple App Store or Google Play).
  • User's active subscription
    • User this option to target only users with/without any active subscriptions or having a specific one.

Adjust traffic level

We do not recommend starting your experiments with 100% of all eligible users. It might be enough to start with 10% to ensure everything works fine and gradually increase your traffic level later.

Launch the setup

Once you have your experiment configured, segmentation, and traffic level set, it's time to start your experiment and proceed to the section with real-time analytics.

Updated 6 months ago

What’s Next