Retention Targeting: From Churn Risk to Incremental Value

A causal decision system that identifies who should receive a retention message—and where campaign spending stops being profitable.

Executive brief

Decision. Do not rank customers by churn risk alone. Prioritize customers whose incremental response × typical basket value justifies the cost of contact.

Why. At a 5% contact budget, an uplift policy finds 147 incremental purchases per 1,000 customers, compared with 32 for churn-risk targeting. But the customers most likely to respond also spend less. Multiplying incremental response by pre-campaign basket value changes the economic ranking and produces the strongest estimated order-value outcome.

Product. The live decision tool lets a campaign owner compare targeting policies, export a customer list, and stress-test profitability against contact-cost and margin assumptions.

Evidence at a glance

Question	Held-out result
Did the randomized SMS campaign change purchase behavior?	+3.3 pp average effect; 95% CI 2.2–4.4 pp
Does behavior beat a recency-only retention rule?	Test PR-AUC 0.672 vs. 0.572
Does uplift beat churn-risk targeting?	Qini 0.0283 ± 0.0017 vs. 0.0220
Who produces more incremental purchases at 5% reach?	Uplift 147 vs. 32 per 1,000
Which policy produces more estimated order value at 5% reach?	Value-aware ₽82K vs. uplift-only ₽28K vs. risk ₽20K per 1,000
Is messaging everyone a safe default?	No—random targeting loses money at every tested budget

Order-value figures are planning estimates in Russian rubles (RUB), not audited realized revenue. On the randomized holdout, they compare the treatment and control means of purchase outcome × pre-campaign average basket, then scale the difference to the selected population. Contact cost and gross margin are explicit assumptions in the sensitivity analysis.

The product problem

Subscription products observe cancellation. Transactional products usually do not: a customer who has left and one who is simply between purchases can look identical. That creates two different questions:

1. Who may not return? A retention prediction problem.
2. Whose behavior will a message change? A causal treatment-effect problem.

A high churn score answers only the first. Retention spending requires the second—and the value of the behavior changed.

Data and experimental design

The project uses the public X5 RetailHero uplift dataset:

• 400,162 loyalty customers
• 45.8 million purchase records across 117 days
• 200,039 customers in a randomized SMS campaign
• 30,006 customers in the untouched application/evaluation sample

Randomization was checked rather than assumed: every pre-campaign feature differs by less than 1% between treatment and control. Train, validation, and test were separated; the test set was opened once for selected models.

Decision pipeline

Layer 1: behavioral segmentation

K-means on twelve scaled behavioral features was compared with Gaussian Mixtures across k=2…10 and an RFM-quantile baseline. Selection balanced separation, segment size, stability, and interpretability rather than optimizing silhouette alone.

Two groups that look similar under standard RFM behave differently: frequent small-basket and occasional large-basket shoppers have a 12-point difference in non-return rate. Purchase rhythm carries information that RFM misses.

Layer 2: retention modeling

Nine models were compared, from recency-only and logistic-regression baselines through bagged trees and boosting libraries. Logistic regression was selected under a rule stated in advance: choose the simpler model when its validation score is within 0.005 of the leader.

CatBoost ranks raw recency first; logistic regression ranks it 22nd because the linear model relies on an engineered recency-to-shopping-rhythm interaction. Both models finish within 0.005 PR-AUC.

Layer 3: uplift and policy evaluation

Four uplift meta-learners were compared across five random seeds. Model selection used Qini against both random ranking and churn-risk ranking—not a single favorable split.

The initial hypothesis was wrong: response did not peak in the middle-risk group. It increased with modeled risk because the campaign population excluded customers inactive for more than 22 days; the “already gone” group was absent by design.

The winning policy depends on the objective. Incremental-response ranking wins on purchase count; incremental response × typical basket value wins on estimated order value.

Application walkthrough

The Streamlit product exposes four decision moments:

1. Executive overview — effect size, uncertainty, control rate, segment economics, and the recommended policy.
2. Build a target list — compare value-aware, uplift-only, and risk-only targeting under a fixed reach.
3. Test the economics — vary contact cost and gross margin to locate the profit-maximizing stopping point.
4. Inspect a customer — review an individual recommendation and its rank.

Reproduce

git clone https://github.com/reetu95/Customer-Retention-Segmentation-Uplift.git
cd Customer-Retention-Segmentation-Uplift
python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
make check
make run

The app uses the compact held-out artifact committed at app/app_data.parquet. The full analysis notebook can reproduce features, model comparisons, and figures; the source dataset downloads through sklift.datasets.fetch_x5().

For the modeling environment, install the additional dependencies with pip install -r requirements-analysis.txt and open notebooks/EDA.ipynb in JupyterLab or Google Colab.

Repository structure

.
├── app/
│   ├── app_data.parquet       # held-out application artifact
│   ├── logic.py               # tested policy and economics calculations
│   └── streamlit_app.py       # decision interface
├── notebooks/
│   └── EDA.ipynb              # analysis, modeling, and evaluation
├── reports/figures/           # recruiter-facing and analytical visuals
├── tests/
│   └── test_logic.py          # unit and data-contract tests
├── .github/workflows/
│   └── quality.yml            # compile and test on every push/PR
├── .streamlit/config.toml     # consistent product theme
├── Makefile
├── requirements.txt          # lightweight app and test environment
└── requirements-analysis.txt # full notebook/modeling environment

Quality controls

• Eight automated unit and data-contract tests
• CI compilation and tests on Python 3.12
• Unique-customer and binary-treatment/outcome checks
• Pure decision functions separated from the Streamlit presentation layer
• Explicit labels for observed effects, modeled risks, and economic assumptions

Limitations

• The 117-day history measures “did not return soon,” not permanent churn.
• The randomized outcome is purchase/no purchase, not realized order value.
• Order-value estimates use pre-campaign average basket as a proxy; realized post-treatment order values are unavailable.
• Contact cost and margin are user-controlled assumptions, not observed fields.
• The data is from a grocery retailer, not a subscription or two-sided marketplace.
• Policy comparisons are offline; a prospective experiment should validate the value-aware policy before production rollout.

What I would test next

Randomize eligible customers across uplift-only, uplift × value, and business-as-usual policies. Pre-register incremental gross margin per eligible customer as the primary metric, with unsubscribe rate, complaint rate, and longer-term purchase frequency as guardrails.

Built with

Python · pandas · scikit-learn · CatBoost · LightGBM · XGBoost · scikit-uplift · Plotly · Streamlit · GitHub Actions

Released under the MIT License.