• Performance impact analysis

  • A performance impact analysis helps you evaluate the impact of Buy with Prime on your ecommerce site, with A/B testing as a cornerstone offering. With a performance impact analysis, you can collect key data points about your ecommerce site and shopper experience and uncover critical insights that can inform key business decisions. 

  • What is an A/B test?

    An A/B test is a type of performance impact analysis. It can help you uncover critical insights that can inform key business decisions. A/B testing compares two versions of a user experience (a control group and a test group) against each other to reveal which performs better based on your test goal. For example, you could test your current shopper experience without Buy with Prime compared to your shopper experience with Buy with Prime. 

  • Requirements

    The only requirements are that you conduct your A/B test with VWO, the third-party service provider supporting Buy with Prime merchants on A/B testing, and start your test within the first 30 days of registering for your free trial with VWO. Beyond these requirements, there are some best practices we recommend to help ensure that you’re getting the most meaningful results in your A/B tests.

    • Although all Buy with Prime merchants can participate in A/B testing, merchants with monthly traffic greater than 10,000 visitors are best suited for A/B testing because they see statistically significant results faster than merchants with lower monthly traffic.
    • Test 100% of the SKUs that you intend to offer Buy with Prime on. For example, if you have 100 SKUs on your site and you plan to activate Buy with Prime on 80% of them, we recommend including all 80 SKUs in the test.
    • Activate Buy with Prime on your most popular SKUs to collect the highest volume of data.
    • Start by testing button placement on your product pages. We recommend placing it below your site’s native “Add to cart” button.
    • Test one thing at a time so that you can easily attribute results and make changes as needed.

    Check out more tips and best practices on the related Buy with Prime blog article.

  • How does A/B test work?

    To conduct an A/B test of Buy with Prime on your site, you need to:

    • Confirm your fit: Merchants with monthly traffic greater than 10,000 visitors are best suited for A/B testing because they see statistically significant results faster than merchants with lower traffic.

    • Launch your A/B test: Work with third-party service provider VWO to A/B test different shopper experiences on select product pages to identify the best-performing version. To sign up for the interest list, go to Performance Impact Analysis interest list and complete the form. After your application is processed, a sales representative will send you an invitation email to start A/B testing Buy with Prime services using VWO.

    • Collect data from randomized transactions: During the test, your site visitors are split between the different shopper experiences, allowing you to collect comprehensive data.

    • Use learnings to boost growth: Evaluate your A/B test results to determine the impact of Buy with Prime on your site and identify ways to improve your shopper experience to help boost your bottom line.

  • Setting up A/B testing for Buy with Prime on Shopify stores

    To set up your ecommerce site for A/B testing Buy with Prime services using VWO you need to:

    1. Create your VWO account
    2. Install the VWO SmartCode and Revenue Code on your site
    3. Select product URLs and launch the test

     

    Note: Before you begin, ensure you have Admin access to the Shopify Admin Panel.

    Important: To sign up for the interest list, go to Performance Impact Analysis interest list and complete the form. After your application is processed, a sales representative will send you an invitation email to start A/B testing Buy with Prime services using VWO.

     

    Create your VWO account

    IMPORTANT: To create a VWO account, you must use a work email address with a domain that matches the testing URL. Don’t use a personal or an agency email address.

    Step 1Open your invitation email, and then choose the link provided in the email to create your VWO account.

    Step 2Enter your work email address, and then choose START TESTING NOW.

    Step 3Enter the required information, and then choose CREATE ACCOUNT.

    After your account is created, you receive an activation email

    Step 4Open the activation email and activate your account.

    IMPORTANT: If you used VWO before, ensure that any existing VWO code is removed from your site before installing the new code. 

     

    Install the VWO SmartCode and Revenue Code on your site

    Step 1Open your VWO dashboard.

    Step 2Choose Settings, and then SmartCode.

    Step 3Copy the SmartCode as shown in the following example:

    Step 4Sign in to your Shopify store as an Admin.

    Step 5On the side menu, choose Sales channels, then Online store, and then Themes.

    Step 6Create a duplicate of the theme file as a backup.

    Step 7Choose the more actions (…) drop-down menu on your theme, and then Edit Code.

    Step 8Select the Themes tab, then choose Current Theme, and then Edit Code.

    Step 9On the side menu, locate and open the theme.liquid file.

    Step 10Paste the SmartCode on the theme.liquid file directly before the closing <head> tag.

    Step 11Choose Save.

    Step 12On the Shopify Admin Panel, choose Settings, and then Checkout.

    Step 13On the Order Status section, locate Additional Scripts.

    Step 14In the Additional Scripts editor, paste the SmartCode as the last line of code.

    Step 15On your VWO dashboard, locate and copy the Revenue code, which looks like the following example:

    {% if first_time_accessed %}
<script type="text/javascript">
window.VWO = window.VWO || [];
{% assign _vis_opt_revenue = subtotal_price | divided_by: 100.0 %}
window._vis_opt_queue = window._vis_opt_queue || [];
window._vis_opt_queue.push(function() {_vis_opt_revenue_conversion('{{ _vis_opt_revenue }}',{allowMultipleConsversions: true});});
window.VWO.push(['track.revenueConversion','{{ _vis_opt_revenue }}']);
</script>
{% endif %}

    Step 16In the Additional Scripts editor, paste the Revenue code directly after the SmartCode that you pasted in Step 14.

    Step 17Choose Save.

    Step 18Verify if the SmartCode is installed correctly on your site by using the URL checker as shown in the following example:

    Step 19Choose Finish.

    Note: If you have any issues during the code installation, contact merchant support.

     

    Select product URLs and launch test

    Step 1Open your VWO dashboard.

    Step 2On the side menu, choose Testing, and then A/B.

    Step 3Select the test that you want to run:

    • Buy with Prime Shopify App Test: Allows you to understand the impact of Buy with Prime on shopper conversion rates when using the Buy with Prime Shopify app. 

    • Buy with Prime Widget Cart + Reviews from Amazon Widget Test: Allows you to understand the impact of Buy with Prime Cart with Reviews from Amazon on shopper conversion rates.

    • Buy with Prime Widget Cart Test: Allows you to understand that impact of Buy with Prime Cart on shopper conversion rates.

    • Buy with Prime Widget + Reviews from Amazon Widget Test: Allows you to understand that impact of Buy with Prime + Reviews from Amazon on shopper conversion rates.

    • Reviews from Amazon Widget Test: Allows you to understand that impact of Reviews from Amazon on shopper conversion rates.

    • Buy with Prime Widget Test: Allows you to understand the impact of Buy with Prime on shopper conversion rates.

    Note: You can run only one test at a time.

    Step 4Choose Settings, and then URL(s).

    Step 5Confirm that the product URLs are correct according to your test type:

    • If you’re testing all products: Include the base URL of your site as shown in the following example:

    • If you’re testing specific products:
      1. Change the URL settings to Advanced as shown in the following example:

      2. In the Included URLs section, add the product URLs that you want to include or exclude as shown in the following example:

    Step 6Choose Save.

    Step 7To start the A/B test on your site, choose Start.

    Your A/B test is up and running. To see test results, contact your sales representative to schedule an end of campaign review.

    Note: If you have any issues setting up A/B testing on your site, contact merchant support.

  • Performance impact analysis FAQ

    Get answers to common questions about performance impact analysis and A/B testing.

    • An A/B test is an accurate and efficient way to determine the impact on shopper conversion when offering Buy with Prime on your site. It minimizes variables that might skew results collected through other methods of analysis, such as seasonality, days of the week, and ongoing promotions. Upon completion of an A/B test, you receive a detailed report on the impact of Buy with Prime on your site and tailored post-analysis recommendations to improve your shopper experience and help boost shopper conversions.

    • VWO collects and stores data associated with the aspects that you’re testing on your site, such as page activity and orders completed, and provides it to Buy with Prime for purposes of analyzing your site's performance and providing recommendations to you on how to leverage Buy with Prime features and services. The data collected is aggregated with data from other merchants that are participating in A/B testing to help us understand and communicate the overall impact of Buy with Prime features and services. VWO doesn’t collect site or business data unrelated to the A/B test, personal identifying information on customers, or order details.

    • Buy with Prime receives data that VWO collects during A/B testing on your site. It’s used in two ways:

      • The Buy with Prime performance impact analysis team collects and reviews the data, and then provides it to your Buy with Prime sales representative to share with you. They also provide results-based recommendations for improving shoppers’ experience using Buy with Prime on your site to help drive greater impact.

      • Buy with Prime may use aggregated, anonymized information collected from participating Buy with Prime merchants to publish recommendations and the latest performance metrics about Buy with Prime.

    • You receive results halfway through each test and at the end of the test. An A/B test typically lasts about 3–5 weeks, which is the average amount of time typically needed to capture statistically significant results. Additional tests on updated versions of your site or experience may be necessary to capture additional learnings and find the best-performing version.

    • Yes, after you have conducted an initial A/B test of Buy with Prime on your site, you can use A/B testing on other aspects of your site or shopper experience

    • You can receive a free nine-month, enterprise-level trial with VWO for A/B testing as long as your first test measures an aspect of Buy with Prime on your site, such as button placement or Buy with Prime badge usage, and is started within the first 30 days of registering for your free trial with VWO.

    Back to top

On this page

show more
arrow
Was this page helpful?
What didn’t work for you?

Thank you! 

Your feedback helps us improve this page.

Was this page helpful?
What didn’t work for you?

Thank you! 

Your feedback helps us improve this page.