CRM and retention

Control group: how to tell a campaign worked

A control group (holdout) is a random part of a segment that does not receive the campaign. The difference in results between the campaign group and the control group shows the true effect (uplift), not the players who would have come anyway.

Yevhen SychovOctober 2026

Formula and example

Effect (uplift) = campaign group result minus control group resultA reactivation offer: 8% of recipients came back, against 3% of the control group. The effect is 5 percentage points, not 8%.

It works the same way in money: deposits and GGR of the campaign group, minus the control group, minus the bonus cost. If the result is below zero, the campaign costs more than it brings in.

Size and split

  • Randomly, not "those who did not open the email" and not "another country". Otherwise the groups differ in more than just the campaign.
  • Enough players: usually 5-10% of the segment, but no fewer than a few hundred people. In small groups the difference can be down to chance.
  • Global holdout: a small share of the base that gets no campaigns at all for months. It shows the contribution of CRM as a whole.

Common mistakes

  • Comparing with last month. Seasonality, traffic and releases move the result more than the campaign does.
  • Crediting the offer with everyone who came back. Some players would have come back without it, and they did not need the bonus.
  • Dropping the control once the campaign "definitely works". The effect changes over time, so the control is needed all the time.
All glossary terms