Your Model Hasn’t Changed. Your Population Has. Here’s How to Detect It

Understanding PSI, VSI, Kolmogorov–Smirnov (KS), and the statistics behind population drift with Python.

A model can be perfectly stable and still become unreliable. The code hasn’t changed. The coefficients are exactly where you left them. The validation metrics looked perfectly respectable when the model was developed. Yet something about its predictions no longer feels quite right.

Perhaps the underlying population has changed.

In credit risk, this could mean that the borrowers applying for loans today have different income profiles, leverage ratios, or credit histories than those represented in the development sample. In machine learning, a recommendation engine might suddenly encounter an entirely different mix of users. This is population drift: a change in the statistical distribution of the observations encountered by a model.

And while there are several ways to measure it, the challenge is rarely just calculating a statistic. It’s understanding what that statistic is actually telling us. Let’s explore the tools commonly used for population stability monitoring, their limitations, and how to implement them in Python.

1. Population Stability Index (PSI):

If you’ve worked in credit risk modelling, you’ve almost certainly encountered PSI.

Population Stability Index measures how much a variable’s distribution has shifted between two populations, typically a reference or development sample and a more recent monitoring sample.

Its formula is:

where:

• (p_i) is the proportion of observations in bin (i) in the reference population.
• (q_i) is the corresponding proportion in the current population.
• (k) is the number of bins.

PSI is closely related to information theory. In fact, it is the sum of the two directed Kullback–Leibler divergences, also known as the Jeffreys divergence.

Consequently, PSI is non-negative and equals zero when the binned distributions are identical.

What do PSI thresholds mean?

The familiar interpretation is:

But here’s where things get interesting. These thresholds are industry conventions, not universal statistical laws.

PSI has no universal sampling-based critical value analogous to a classical hypothesis test. A PSI of 0.12 calculated from 200 observations should not necessarily trigger the same response as a PSI of 0.12 calculated from 200,000 observations.

The 2025 Global Credit Data representativeness guidelines also present these familiar thresholds as reference values rather than prescribed standards.

The binning problem

Suppose we calculate PSI using five bins and obtain 0.08.

Now we use 20 bins on exactly the same observations and obtain 0.19.

Did the population suddenly become less stable?

Of course not. We changed the way we measured it.

PSI is sensitive to the binning strategy, particularly the number of bins, bin boundaries, and treatment of sparse categories.

A sensible approach is to define bins using the reference population and reuse those boundaries consistently during monitoring. For example, reference deciles are common in credit risk monitoring.

Recalculating bins separately for each monitoring period defeats the purpose of maintaining a consistent reference.

2. Variable Stability Index (VSI): Same Family, Different Focus

You may also encounter the term Variable Stability Index, or VSI.The terminology is not entirely standardized. In many credit risk monitoring frameworks, VSI refers to applying the same PSI-style calculation to individual explanatory variables, while PSI is reserved for the overall score or model output.

Under that convention:

• PSI: Has the distribution of model scores or predicted risk changed?
• VSI: Have the distributions of individual predictors changed?

Consider a probability of default model using borrower age, debt-to-income ratio, credit utilization, and delinquency history.

The overall score distribution might remain reasonably stable while credit utilization experiences a substantial shift.

A score-level PSI alone could miss that underlying change, particularly when shifts across predictors offset one another.

Monitoring individual variables provides a more granular view of the changing population.

However, there is an important limitation: univariate stability does not guarantee multivariate stability.

Two variables can retain identical marginal distributions while their correlation changes dramatically.

A model relying on their interaction may deteriorate even though both variables report perfectly acceptable VSI values.

That is why a useful monitoring framework must eventually look beyond individual-variable drift.

3. Kolmogorov–Smirnov: Comparing Entire Distributions

The two-sample Kolmogorov–Smirnov (KS) statistic measures the maximum absolute difference between two empirical cumulative distribution functions.

Here, (F_n(x)) and (G_m(x)) represent the empirical CDFs of the reference and current populations.

Unlike PSI, KS does not require binning.

It examines the greatest vertical separation between the two cumulative distributions.

An important distinction for credit risk practitioners

The KS statistic appears in two different contexts.

In credit scoring, KS is commonly used as a discrimination metric, measuring the separation between the score distributions of defaulters and non-defaulters.

In population drift monitoring, the two-sample KS statistic compares the distribution of a variable across two populations or time periods.

The underlying mathematical concept is similar, but the populations being compared — and therefore the interpretation — are different.

The sample-size problem

A KS test returns both a test statistic and a p-value.

With sufficiently large samples, even a small distributional difference can produce a very small p-value.

Consider a model monitoring 500,000 observations every month. A statistically significant result may represent a distributional change too small to matter operationally.

Conversely, a small sample may have insufficient power to detect a meaningful shift.

So a p-value should not be your only decision criterion.

Statistical significance is not the same as practical significance.

Another consideration: the classical two-sample KS p-value assumes independent observations from continuous distributions. Discrete variables, many ties, or repeated observations can complicate its interpretation.

4. Beyond PSI and KS: Other Useful Drift Metrics

PSI and KS are valuable, but they do not capture every dimension of distributional change.

Wasserstein Distance

Wasserstein distance, sometimes called the Earth Mover’s Distance, measures the cost of transforming one distribution into another.

For one-dimensional distributions:

One attractive property is interpretability: its value is expressed in the original variable’s units.

If a credit utilization ratio is measured between zero and one, the Wasserstein distance is measured on that same scale.

Unlike KS, which focuses on the maximum CDF separation, Wasserstein incorporates the distance over the distribution.

However, raw Wasserstein distances should not be compared indiscriminately across variables with different measurement scales.

Jensen–Shannon Divergence

Jensen–Shannon divergence provides a symmetric, bounded alternative to KL divergence.

where:

Using natural logarithms, JS divergence ranges from zero to (\ln(2)).

Its square root, Jensen–Shannon distance, is also commonly used in drift-monitoring libraries.

It is particularly useful when comparing categorical distributions or binned continuous features, although bin selection still matters.

Chi-Squared Test

For categorical predictors such as employment type, industry classification, or geographic region, a chi-squared homogeneity test can assess changes in category proportions.

But once again, sample size matters.

Sparse categories can also violate the usual large-sample approximations. In such cases, category aggregation or an appropriate alternative test may be needed.

Where (O_{ij}) and (E_{ij}) are the observed and expected frequencies in cell ((i,j)) of the contingency table.

5. Let’s Implement Population Drift Monitoring in Python

We’ll simulate two populations representing a model’s development sample and a later monitoring period.

Assume the current population has somewhat higher credit utilization and a slightly different customer mix.

First, install the dependencies:

pip install numpy pandas scipy matplotlib

Now generate the samples:

import numpy as np
import pandas as pd
from scipy.stats import (
    ks_2samp,
    wasserstein_distance,
    chi2_contingency,
    entropy
)
rng = np.random.default_rng(42)
n = 10000
reference = pd.DataFrame({
    "utilization": rng.beta(2, 5, n),
    "segment": rng.choice(
        ["Retail", "SME", "Corporate"],
        n,
        p=[0.60, 0.30, 0.10]
    )
})
current = pd.DataFrame({
    "utilization": rng.beta(2.5, 4.5, n),
    "segment": rng.choice(
        ["Retail", "SME", "Corporate"],
        n,
        p=[0.48, 0.37, 0.15]
    )
})

Building a reusable PSI function

For a continuous variable, we’ll use quantile bins derived exclusively from the reference sample.

We’ll also include bins for missing values and use small pseudocounts to prevent logarithms of zero.

def psi_numeric(ref, cur, bins=10, alpha=0.5):
    ref = pd.Series(ref, dtype=float)
    cur = pd.Series(cur, dtype=float)
    ref_valid = ref.dropna()
    cur_valid = cur.dropna()
    if ref_valid.empty or cur_valid.empty:
        raise ValueError("Non-missing values required.")
    edges = np.unique(
        np.quantile(
            ref_valid,
            np.linspace(0, 1, bins + 1)
        )
    )
    # Expand tails to capture values outside
    # the reference population's range.
    if len(edges) < 2:
        value = edges[0]
        edges = np.array([-np.inf, value, np.inf])
    else:
        edges[0] = -np.inf
        edges[-1] = np.inf
    ref_counts = np.histogram(ref_valid, edges)[0]
    cur_counts = np.histogram(cur_valid, edges)[0]
    # Treat missingness as an additional category.
    ref_counts = np.append(
        ref_counts, ref.isna().sum()
    )
    cur_counts = np.append(
        cur_counts, cur.isna().sum()
    )
    # Identical smoothing convention for both.
    p = (ref_counts + alpha) / (
        len(ref) + alpha * len(ref_counts)
    )
    q = (cur_counts + alpha) / (
        len(cur) + alpha * len(cur_counts)
    )
    return float(np.sum((p - q) * np.log(p / q)))

The smoothing avoids infinite values when a bin is empty. It is a practical numerical choice, not a universal PSI standard; the selected smoothing approach should be documented.

For categorical variables, we can apply the same calculation using a common set of categories:

def psi_categorical(ref, cur, alpha=0.5):
    ref = pd.Series(ref).fillna("__MISSING__")
    cur = pd.Series(cur).fillna("__MISSING__")
    categories = ref.unique().tolist()
    categories += [
        x for x in cur.unique()
        if x not in categories
    ]
    p_count = ref.value_counts().reindex(
        categories, fill_value=0
    ).to_numpy()
    q_count = cur.value_counts().reindex(
        categories, fill_value=0
    ).to_numpy()
    k = len(categories)
    p = (p_count + alpha) / (len(ref) + alpha * k)
    q = (q_count + alpha) / (len(cur) + alpha * k)
    return float(np.sum((p - q) * np.log(p / q)))

Comparing multiple metrics

x = reference["utilization"]
y = current["utilization"]
ks = ks_2samp(x, y)
psi_value = psi_numeric(x, y)
wasserstein = wasserstein_distance(x, y)
# Jensen-Shannon divergence using reference bins
edges = np.quantile(x, np.linspace(0, 1, 11))
edges = np.unique(edges)
edges[0], edges[-1] = -np.inf, np.inf
p = np.histogram(x, bins=edges)[0].astype(float)
q = np.histogram(y, bins=edges)[0].astype(float)
p /= p.sum()
q /= q.sum()
m = 0.5 * (p + q)
js_divergence = 0.5 * (
    entropy(p, m) + entropy(q, m)
)
print(f"PSI: {psi_value:.4f}")
print(f"KS statistic: {ks.statistic:.4f}")
print(f"KS p-value: {ks.pvalue:.4g}")
print(f"Wasserstein distance: {wasserstein:.4f}")
print(f"JS divergence: {js_divergence:.4f}")
print(
    "Categorical VSI:",
    round(
        psi_categorical(
            reference["segment"],
            current["segment"]
        ), 4
    )
)

These statistics measure different aspects of drift. They should not be interpreted as interchangeable scores.

Visualizing the distribution shift

import matplotlib.pyplot as plt
plt.figure(figsize=(9, 5))
plt.hist(
    x, bins=40, density=True,
    alpha=0.5, label="Reference"
)
plt.hist(
    y, bins=40, density=True,
    alpha=0.5, label="Current"
)
plt.xlabel("Credit Utilization")
plt.ylabel("Density")
plt.title("Population Drift in Credit Utilization")
plt.legend()
plt.tight_layout()
plt.show()

Often, the visualization is as important as the summary statistic. A single number cannot tell you whether the difference originates in the center of the distribution, the tails, or a newly emerging subgroup.

6. What Drift Statistics Cannot Tell You

Here’s the uncomfortable truth: a population can drift without model performance deteriorating, and model performance can deteriorate without obvious population drift.

There are several reasons.

First, covariate drift does not automatically imply that the relationship between predictors and outcomes has changed.

Second, concept drift can change that relationship even when the predictor distributions remain relatively stable.

Third, changes in correlations or interactions can escape univariate monitoring.

Finally, in credit risk, realized defaults may arrive with substantial delays. Population drift indicators often become available well before reliable performance metrics.

Therefore, a mature monitoring framework should combine:

• Input stability: PSI/VSI, KS, Wasserstein, and categorical distribution checks.
• Data quality: Missingness, outliers, unseen categories, and data pipeline changes.
• Model output stability: Score distributions, approval rates, risk-grade migration, and concentration.
• Model performance: Discrimination, calibration, and observed-versus-expected outcomes when reliable labels become available.
• Multivariate monitoring: Correlation shifts, interaction diagnostics, or classifier-based drift detection.

A useful addition is an adversarial validation model trained to distinguish reference observations from current observations. If it can separate them reliably on held-out data, the joint feature distributions may have changed, even when individual variables appear stable.

7. Stop Treating Thresholds as Verdicts

PSI above 0.25 does not automatically mean a model must be recalibrated.

Likewise, PSI below 0.10 does not prove that a population is stable in every meaningful sense.

A better approach is to establish monitoring thresholds using historical data, expected sampling variability, business context, and observed relationships between drift and model performance.

Where appropriate, bootstrap or simulation-based reference distributions can help distinguish routine variability from unusual changes.

Monitoring should also be performed consistently across comparable time windows and portfolio segments. Seasonality, changing sample sizes, and product mix can otherwise create misleading alerts.

The real purpose of population monitoring is not to produce a traffic-light dashboard.

It is to identify changes early enough to investigate them, understand their consequences, and make informed decisions.

A drift statistic should start a conversation, not end one.

And that, perhaps, is the most important distinction between calculating population stability metrics and actually managing model risk.

References and Further Reading

1. Global Credit Data — Guidelines on Assessing Representativeness (2025)
2. SciPy — Two-Sample Kolmogorov–Smirnov Test
3. Evidently AI — Data Drift Detection Deep Dive
4. NannyML — Univariate Drift Detection
5. MathWorks — Population Stability Index