Performance impact analysis

An A/B test is a type of performance impact analysis. It can help you uncover critical insights that can inform key business decisions. A/B testing compares two versions of a user experience (a control group and a test group) against each other to reveal which performs better based on your test goal. For example, you could test your current shopper experience without Buy with Prime compared to your shopper experience with Buy with Prime.

The only requirements are that you conduct your A/B test with VWO, the third-party service provider supporting Buy with Prime merchants on A/B testing, and start your test within the first 30 days of registering for your free trial with VWO. Beyond these requirements, there are some best practices we recommend to help ensure that you’re getting the most meaningful results in your A/B tests.

• Although all Buy with Prime merchants can participate in A/B testing, merchants with monthly traffic greater than 10,000 visitors are best suited for A/B testing because they see statistically significant results faster than merchants with lower monthly traffic.
• Test 100% of the SKUs that you intend to offer Buy with Prime on. For example, if you have 100 SKUs on your site and you plan to activate Buy with Prime on 80% of them, we recommend including all 80 SKUs in the test.
• Activate Buy with Prime on your most popular SKUs to collect the highest volume of data.
• Start by testing button placement on your product pages. We recommend placing it below your site’s native “Add to cart” button.
• Test one thing at a time so that you can easily attribute results and make changes as needed.

Check out more tips and best practices on the related Buy with Prime blog article.

To conduct an A/B test of Buy with Prime on your site, you need to:

• Confirm your fit: Merchants with monthly traffic greater than 10,000 visitors are best suited for A/B testing because they see statistically significant results faster than merchants with lower traffic.
• Launch your A/B test: Work with third-party service provider VWO to A/B test different shopper experiences on select product pages to identify the best-performing version. To sign up for the interest list, go to Performance Impact Analysis interest list and complete the form. After your application is processed, a sales representative will send you an invitation email to start A/B testing Buy with Prime services using VWO.
• Collect data from randomized transactions: During the test, your site visitors are split between the different shopper experiences, allowing you to collect comprehensive data.
• Use learnings to boost growth: Evaluate your A/B test results to determine the impact of Buy with Prime on your site and identify ways to improve your shopper experience to help boost your bottom line.

To set up your ecommerce site for A/B testing Buy with Prime services using VWO you need to:

1. Create your VWO account
2. Install the VWO SmartCode and Revenue Code on your site
3. Select product URLs and launch the test

Note: Before you begin, ensure you have Admin access to the Shopify Admin Panel.

Important: To sign up for the interest list, go to Performance Impact Analysis interest list and complete the form. After your application is processed, a sales representative will send you an invitation email to start A/B testing Buy with Prime services using VWO.

IMPORTANT: To create a VWO account, you must use a work email address with a domain that matches the testing URL. Don’t use a personal or an agency email address.

1. Open your invitation email, and then choose the link provided in the email to create your VWO account.
2. Enter your work email address, and then choose START TESTING NOW.
   

   

3. Enter the required information, and then choose CREATE ACCOUNT.
   

   

   After your account is created, you receive an activation email
4. Open the activation email and activate your account.

IMPORTANT: If you used VWO before, ensure that any existing VWO code is removed from your site before installing the new code.

1. Open your VWO dashboard.
2. Choose Settings, and then SmartCode.
3. Copy the SmartCode as shown in the following example:
   

   

4. Sign in to your Shopify store as an Admin.
5. On the side menu, choose Sales channels, then Online store, and then Themes.
6. Create a duplicate of the theme file as a backup.
7. Choose the more actions (…) drop-down menu on your theme, and then Edit Code.
8. Select the Themes tab, then choose Current Theme, and then Edit Code.
9. On the side menu, locate and open the theme.liquid file.
10. Paste the SmartCode on the theme.liquid file directly before the closing <head> tag.
11. Choose Save.
12. On the Shopify Admin Panel, choose Settings, and then Checkout.
13. On the Order Status section, locate Additional Scripts.
14. In the Additional Scripts editor, paste the SmartCode as the last line of code.
15. On your VWO dashboard, locate and copy the Revenue code, which looks like the following example:
    

    {% if first_time_accessed %}
     <script type="text/javascript"> 
    window.VWO = window.VWO || []; 
    {% assign _vis_opt_revenue = subtotal_price | divided_by: 100.0 %} 
    window._vis_opt_queue = window._vis_opt_queue || []; window._vis_opt_queue.push(function() {_vis_opt_revenue_conversion('{{ _vis_opt_revenue }}',{allowMultipleConsversions: true});}); window.VWO.push(['track.revenueConversion','{{ _vis_opt_revenue }}']); 
    </script> 
    {% endif %}

16. In the Additional Scripts editor, paste the Revenue code directly after the SmartCode that you pasted in Step 14.
17. Choose Save.
18. Verify if the SmartCode is installed correctly on your site by using the URL checker as shown in the following example:
    

    

19. Choose Finish.

Note: If you have any issues during the code installation, contact merchant support.

1. Open your VWO dashboard.
2. On the side menu, choose Testing, and then A/B.
3. Select the test that you want to run:
   
   • Buy with Prime Shopify App Test: Allows you to understand the impact of Buy with Prime on shopper conversion rates when using the Buy with Prime Shopify app.
   • Buy with Prime Widget Cart + Reviews from Amazon Widget Test: Allows you to understand the impact of Buy with Prime Cart with Reviews from Amazon on shopper conversion rates.
   • Buy with Prime Widget Cart Test: Allows you to understand that impact of Buy with Prime Cart on shopper conversion rates.
   • Buy with Prime Widget + Reviews from Amazon Widget Test: Allows you to understand that impact of Buy with Prime + Reviews from Amazon on shopper conversion rates.
   • Reviews from Amazon Widget Test: Allows you to understand that impact of Reviews from Amazon on shopper conversion rates.
   • Buy with Prime Widget Test: Allows you to understand the impact of Buy with Prime on shopper conversion rates.

   Note: You can run only one test at a time.

4. Choose Settings, and then URL(s).
5. Confirm that the product URLs are correct according to your test type:
   
   • If you’re testing all products: Include the base URL of your site as shown in the following example:
     

     

   • If you’re testing specific products:
     
     1. Change the URL settings to Advanced as shown in the following example:
        

        

     2. In the Included URLs section, add the product URLs that you want to include or exclude as shown in the following example:
        

        

6. Choose Save.
7. To start the A/B test on your site, choose Start.

Your A/B test is up and running. To see test results, contact your sales representative to schedule an end of campaign review.
Note: If you have any issues setting up A/B testing on your site, contact merchant support.

Get answers to common questions about Buy with Prime performance impact analysis.

For other questions, go to FAQ library.