Week 7 – Model Validation and Bayesian Analysis

07 Aug 2026 - Sneha Dharmeche - climate risk, data science, feature engineering, NHS ERIC

Project overview

This week I focused on improving my predictive models and validating the quality of the ERIC estate dataset and peforming Bayesian modelling.

One of my main tasks was developing machine learning models to predict NHS backlog maintenance costs. I trained both Random Forest and XGBoost regression models and evaluated their performance using MAE, RMSE and R². After discussing the results with my supervisor, I explored ways of improving the models by investigating the distribution of the target variable.

I found that backlog maintenance costs were highly skewed, with a small number of sites having extremely large costs compared to most NHS sites. To better understand this, I explored the distribution using summary statistics and quantiles, which showed the presence of several very large outliers. I then applied a log transformation to the target variable, which did not improve performance unfortunately :(

I also began preparing the data for Bayesian modelling using the brms package in R to investigate the factors associated with NHS estate backlog costs. After preparing and cleaning the dataset, I compared different model specifications and found that a simpler model without missing-data indicator variables produced much better convergence diagnostics (Rhat = 1.00 for all parameters) and higher effective sample sizes, which suggests reliable posterior estimates.

The results suggested that age vulnerability score and gross internal floor area were the strongest positive predictors of backlog costs, with their 95% credible intervals not crossing zero. In contrast, variables such as flood occurrences and gas and water consumption showed little evidence of a clear association after accounting for other estate characteristics.

I also found that backlog costs increased across reporting years compared with the 2021/22 baseline, suggesting changes over time in estate condition or reporting.

Through this process, I developed a much better understanding of Bayesian modelling concepts, including priors, posteriors, credible intervals, posterior predictive checks and convergence diagnostics such as Rhat.

Highlight of the week

I think the highlight of my week was treating myself to an end-of-week sweet treat on Friday, a cupcake from Waitrose during my lunch break (and yes, it tasted as good as it looked!).