Over the past few weeks, I have been writing about four topics that, at first glance, may seem quite different: confidence intervals, reparameterization, neural-network symmetry, and GenAI model validation. But there is a common thread connecting all four.
Models are often more subtle than the outputs we see.
A confidence interval looks like a simple range. A neural network looks like a set of learned weights. A stochastic computation looks like something we should simply differentiate through. A GenAI application looks like a model producing an answer.
In each case, however, the interesting part begins when we ask: What is actually happening underneath?
I explored these questions in four recent articles published in Towards Data Science.
1. Reparameterization Tricks: Variance Reduction by Smarter Gradients
One of the beautiful ideas in modern machine learning is that sometimes the best way to solve an optimization problem is not to change the objective, but to change how we represent the randomness inside it.
That is the idea behind the reparameterization trick.
Suppose we want to differentiate an expectation involving a random variable. Sampling inside the computation graph makes gradient estimation difficult and can produce noisy estimators.
Reparameterization rewrites the stochastic variable in terms of a deterministic transformation and a separate source of randomness. The canonical example is:
z = μ + σε, where ε ~ N(0,1)
Instead of sampling directly from a distribution whose parameters we are trying to learn, we move the randomness into ε. This allows gradients to flow through μ and σ using ordinary differentiation.
What I find particularly interesting is that this is not merely a mathematical convenience. It illustrates a broader principle in machine learning:
The way we formulate a problem can dramatically affect how efficiently we can learn from it.
In the article, I explore why reparameterized or pathwise gradients can have much lower variance than score-function estimators, where the technique works, and why it became foundational to methods such as Variational Autoencoders.
👉 Read: Reparameterization Tricks: Variance Reduction by Smarter Gradients
2. The 95% Illusion: Why Your Confidence Interval Isn’t What You Think It Is
This article started from something much more familiar.
Most people who have worked with statistics have encountered the sentence:
“The parameter has a 95% probability of lying inside the 95% confidence interval.”
It sounds completely reasonable.
It is also not the frequentist interpretation of a confidence interval.
Once an interval has been calculated from observed data, the parameter is either inside it or it isn’t. The 95% refers to the long-run coverage of the procedure used to construct the intervals, not a probability assigned to the particular parameter after seeing the data.
That distinction can feel philosophical until it starts affecting actual decisions.
It becomes especially important when comparing frequentist confidence intervals with Bayesian credible intervals. The two approaches can use similar-looking intervals while answering fundamentally different questions about uncertainty.
This is one of the things I enjoy most about statistics: concepts we routinely use can become surprisingly deep when we stop and ask what they really mean.
👉 Read: The 95% Illusion: Why Your Confidence Interval Isn’t What You Think It Is
3. The Symmetry That Breaks Neural Network Averaging
Now consider a seemingly straightforward experiment.
Train the same neural-network architecture twice using different random seeds.
Both models converge. Both perform well.
So what happens if we average their weights?
Intuition might suggest that the result should perform somewhere between the two models—or perhaps even improve through averaging.
Sometimes, it performs terribly.
The culprit is a fascinating structural property of neural networks called permutation symmetry.
Hidden neurons do not have intrinsic identities. Two networks can represent essentially the same function while arranging their hidden units differently.
Imagine two spreadsheets containing the same information but with their columns rearranged. Averaging column one in the first spreadsheet with column one in the second makes little sense unless the columns have first been aligned.
Something similar happens when we naively average neural-network parameters.
This seemingly theoretical property has important consequences for areas such as model merging, weight averaging and the geometry of neural-network loss landscapes.
It is also another reminder that parameters themselves do not always mean what we intuitively think they mean.
👉 Read: The Symmetry That Breaks Neural Network Averaging
4. The Model Validation Playbook for GenAI: Lessons from Banking
The fourth article moves from mathematical properties of models to a question that is becoming increasingly important in practice:
How do we actually validate a Generative AI system?
Traditional model validation grew around predictive models with relatively well-defined inputs, outputs and performance metrics.
You can calculate AUC. You can examine calibration. You can perform backtesting. You can compare predicted values with observed outcomes.
GenAI breaks several of these assumptions.
The output may be non-deterministic.
There may be no single ground truth.
The underlying foundation model may have been developed by a third party.
A system can contain retrieval, prompts, guardrails, orchestration and external tools in addition to the model itself.
And a perfectly fluent answer can still be wrong.
Coming from the world of banking model risk, I wanted to explore which parts of traditional model validation remain useful and which need to evolve.
My central argument is that for GenAI, the unit of validation increasingly needs to become the entire system rather than just the underlying model.
That means thinking about risk tiering, outcome-based testing, hallucination and factuality, robustness, retrieval quality, human oversight, monitoring and silent changes in system behaviour.
Banking happens to provide a useful starting point because it has spent decades developing frameworks for independently challenging models where the consequences of being wrong can be significant.
The question now is how those principles should evolve for systems whose behaviour is probabilistic, contextual and increasingly agentic.
👉 Read: The Model Validation Playbook for GenAI: Lessons from Banking
What connects these four articles?
Writing these pieces made me realise that they are really variations of the same question:
What assumptions are hiding behind the tools we use?
A 95% confidence interval is not quite the probability statement we instinctively want it to be.
A stochastic gradient can behave very differently depending on how randomness is represented.
Two neural networks implementing essentially the same function can have very different-looking parameter vectors.
And a GenAI application cannot necessarily be validated by treating it like the predictive models that came before it.
There is a temptation in data science to move quickly from technique to implementation.
Sometimes the more interesting work begins by going in the opposite direction—slowing down and asking why the technique works at all, what assumptions make it valid, and where those assumptions might fail.
That is the direction I have been enjoying exploring in my recent writing.
If any of these topics interest you, the full articles are available on Towards Data Science:
Reparameterization Tricks: Variance Reduction by Smarter Gradients
https://towardsdatascience.com/reparameterization-tricks-variance-reduction-by-smarter-gradients/
The 95% Illusion: Why Your Confidence Interval Isn’t What You Think It Is
https://towardsdatascience.com/the-95-illusion-why-your-confidence-interval-isnt-what-you-think-it-is/
The Symmetry That Breaks Neural Network Averaging
https://towardsdatascience.com/the-symmetry-that-breaks-neural-network-averaging/
The Model Validation Playbook for GenAI: Lessons from Banking
https://towardsdatascience.com/the-model-validation-playbook-for-genai-lessons-from-banking/
More explorations in statistics, machine learning, model risk and AI to come.