A/B Test Design Pipeline — AI for Fintech Lab

The Problem

A business team wants to test something: a new offer, a different rate, a changed flow. They pull a customer list and split it in half. That split is usually where the test breaks.

Splitting randomly across the whole base can leave one group with more customers from a specific profession, region, or education level than the other. When the results come in, nobody can tell whether the difference came from the test or from the composition of the groups.

This pipeline handles the part that comes before the test runs. It takes a raw customer base and returns two balanced groups, ready to use, with the sample size backed by power analysis, stratified randomization keeping both groups comparable, and a statistical check proving the split was fair.

What It Delivers

Two group files

Control and treatment exported as CSV, ready to hand off to the business team without further processing.

Sample size

Power analysis determines the minimum number of customers per group needed to detect the declared effect, and reports whether the available population is enough.

Balance proof

Every stratification column goes through a t-test or chi-square test after randomization, with a pass or fail result per variable.

Plain text report

A summary written for the business team, with the numbers that matter and no statistical jargon left unexplained.

Key Terms

Stratification

Splitting proportionally within each segment, so both groups keep the same composition as the original population.

Statistical power

The probability of detecting a real effect when one exists. 80% is the standard convention.

Minimum detectable effect

The smallest change the test is designed to catch. Smaller effects require larger samples.

Pipeline Overview

Fig. 1 · Population profiling and stratification check, then sample size, randomization and balance validation, ending in the exported groups and the experiment registry.

Key Decisions

Stratified over simple random

A simple split can leave one group over-represented in a segment purely by chance. Stratifying by the declared columns guarantees proportional composition in both groups from the start, rather than hoping randomness works out.

Sample size calculated, not assumed

Power analysis based on the declared confidence level, statistical power, and minimum detectable effect. The pipeline reports whether the available population is enough, and by how much it falls short when it is not.

Balance validated statistically

Every stratification column goes through a t-test or chi-square test after randomization. The PCA scatter plot is a complementary visual confirmation, not the validation itself.

Numerical columns must be grouped

Declaring a numerical column like age creates one segment per unique value, fragmenting the population into hundreds of tiny groups. The pipeline detects this and warns before the split happens.

Every experiment recorded

Each run is saved to a local SQLite database with its parameters and results, so past experiments can be reviewed without rerunning anything.

Repository

expd-001-ab-test-design-pipeline github.com →

expd-001-ab-test-design-pipeline/
├── README.md
├── requirements.txt
├── .gitignore
├── config/
│ ├── __init__.py
│ └── ab_test_config.py
├── src/
│ ├── __init__.py
│ ├── eda.py
│ ├── stratification_check.py
│ ├── sample_size.py
│ ├── randomization.py
│ ├── balance_validation.py
│ ├── export.py
│ └── registry.py
├── data/
│ └── bank_marketing.csv
├── outputs/
│ ├── group_a.csv
│ ├── group_b.csv
│ ├── experiment_summary.txt
│ └── group_overlap.png
├── db/
│ └── experiments.db
└── notebooks/
└── quickstart.ipynb

Stack: statsmodels, SciPy, scikit-learn, pandas, matplotlib. Python 3.13. Uses the Bank Marketing dataset from a Portuguese banking institution, publicly available on Kaggle and originally from the UCI Machine Learning Repository.