An Alternative to A/B Testing: Change Point Detection

Why Change Point Detection Might be Right for You

While A/B testing is the gold standard for measuring the impact of changes, it’s not always practical. Maybe your site doesn’t have enough traffic to generate statistically significant results, or perhaps you can only modify a single version of a page without the ability to serve different experiences to different users. In these cases, change point detection can provide a data-driven alternative. By analyzing historical engagement trends, Pettitt’s test can help identify when a significant shift occurred, allowing you to infer whether a content change had an impact—even without running a traditional experiment.

What is Pettitt's Test?

Pettitt's test is a statistical method that can detect significant changes in a time series of data. It works by comparing the mean or median of the data before and after each point in time, and identifying the point where the difference is most significant. This is the “change point” where something likely caused the data to shift.

When applied to metrics like conversion rate, click-through rate, or engagement over time on a web page, Pettitt's test can identify if and when a change to the page had a significant impact. While it doesn't definitively prove causality, it provides strong evidence that the change made a difference.

Advantages of Pettitt's Test

One advantage of Pettitt's test over A/B testing is that you can use it on a single version of a page. You don't need to split traffic between a control and variant. This makes it useful for lower-traffic pages or when you can only change the live version due to technical or other constraints. It can also serve as a safety net, allowing you to check if a change you made had unintended consequences even if you didn't or couldn't run an A/B test.

Another powerful aspect of Pettitt's test is that you can use it to detect multiple change points by splitting the data. After identifying an initial change point, you can separate the data at that point and run the test again on each subset. This can reveal if there were any additional significant changes after the first one.

For example, imagine you made a major update to your homepage and want to know its impact. You could use Pettitt's test to find the most significant change point in your conversion rate time series after the update. If you find one, you can be fairly confident the update had an effect. You could then split the data at that change point and run the test separately on the data before and after it. If you find additional change points, this could indicate that the update had ripple effects over time, or that other factors came into play later on.

Limitations of Pettitt's Test

Pettitt's test is not a complete replacement for A/B testing. It doesn't give you the same level of control, and you can't use it to compare two different versions of a page or feature. But when A/B testing isn't feasible, Pettitt's test is a valuable tool to have in your data analysis toolkit. Combined with other methods like user research and heuristic analysis, it can help you keep improving your site or app even when you're constrained in the kinds of tests you can run.

Using Pettitt's Test with contentEnhance

With contentEnhance, you can easily apply Pettitt's test to key engagement metrics for your pages, such as conversion rate, time on page, and bounce rate.

Once you've connected your analytics data to contentEnhance and chosen the pages you want to analyze, the tool will automatically run Pettitt's test to identify potential change points. This allows you to quickly see if and when changes you've made to your content have had a significant impact, without needing to set up a separate A/B test.

contentEnhance also makes it easy to dive deeper by selecting new date ranges, which will rerun the analysis. This can uncover additional change points that may have been masked in the overall data, giving you a more granular view of how your page has evolved over time.

By combining Pettitt's test with contentEnhance's other features like AI-powered content optimization ideas and the ability to track the impact of your changes, you have a comprehensive toolkit for continuously improving your key pages and achieving your engagement goals. Even if you're already running A/B tests, Pettitt's test in contentEnhance can provide an additional layer of insight to help you understand and act on your performance data.