Skip to main content
    For the best learning experience, please log in
    Best Practice
    3 min read

    Best practices for A/B testing

    Follow these A/B testing tips so that you conduct effective experiments and gain deeper insights about your customers.

    Thanks for watching; now, don’t stop there! Continue reading to learn even more best practices for A/B testing.

    Best practice #1

    Manage your test based on audience size

    One size doesn’t fit all when it comes to testing, but try your best to experiment with a large testing audience so that you gain enough data for your results to be meaningful.

    That said, 2 things will be affected by the size of your audience:

    Team looking at their audience test size.
    • Number of test variations
      When you have significant content changes to test or a smaller audience test pool, you should run fewer content variations (e.g., just 2). However, if you implement a small change in your test or if your audience size is very large, you may increase the number of variations you test (e.g., 3 or 4 variations). As a general rule of thumb, we recommend testing no more than 4 variations at a time, as any more than that can make it difficult to judge what drives significant impact.
    • Metrics to track
      At the start of your test, you will choose a key performance indicator, like click or conversion rate, to determine your A/B test. Remember that you will have more clicks on every email than conversions, so you’ll get more data per recipient when looking at clicks. If you are worried that the size of your audience will lead to a lack of statistical significance in testing, then focus on those metrics that see the highest engagement overall.
    Best practice #2

    Wait enough time before evaluating performance

    Our advice: be patient. Testing takes time, and Klaviyo will let you know when an A/B test is complete. It could take 2 hours or 2 weeks depending on the scale of your test, so wait until Klaviyo assures you that the results are ready for you.

    This advice is twofold:

    • Do not edit a running test: as you’ll interfere with the variations and invalidate the results.
    • Do not stop a test: Allow Klaviyo to determine statistical significance and, thus, the winning variation. If you pause the test early and assume there’s a clear winner, then you may risk not allowing enough time for full audience engagement and your results may not be accurate.
    An image of a clock.
    Best practice #3

    Run smarter tests with AI

    Klaviyo’s AI-powered A/B testing can help you improve your content, and even gain inspiration for what to test out in your content.

    Here are some tools you can use today:

    • Subject line AI: Quickly generate subject lines based on your brand and message goals; you can even test the different AI-generated options in your experiments. Learn how to use the subject line AI.
    • Smart Send Time: Improve the performance of your email campaigns by running a Smart Send Time test. This finds the best time to send your campaign content to your specific audience. Learn how to use Smart Send Time.
    • Personalized campaigns AI: This advanced A/B test learns what email or SMS campaign variation works for each person, helping you increase 1:1 personalization at scale (available for accounts with 400k+ profiles). Learn how to use personalized campaign AI.
    Best practices for A/B testing
    Back to all resources