SaferWorldbyDesign: Coverage Hierarchies of Post-Hoc Ensemble Models Optimized for Different Goal Functions

Speakers: Eva Bay Wedebye & Nikolov G. Ikolai (DTU National Food Institute)  

Hyperparameter optimization is a procedure where models with different combinations of hyperparameters are trained and their performance assessed, with the purpose of finding an optimum model according to pre-defined performance criteria [1]. We have developed a modelling approach where hyperparameter optimization is used to generate, instead of a single best model, multiple models with different applicability domains (AD) sizes and with different prediction accuracy [2]. As it is well-known that there is a trade-off between applicability domain size and prediction accuracy, a hierarchy of models with increasing AD size can be valuable in providing models with high accuracy as well as others where some accuracy is sacrificed to obtain higher coverage. 

We have implemented this approach in DanishQSAR, our new software for binary classification QSAR modelling. The various methods and options available in the software are checked and combined automatically, resulting in models optimized for sensitivity, specificity or balanced accuracy. To this end, we have developed a version of cross-validation-based grid search and post-hoc ensemble modelling. This enables the system to efficiently generate a large and diverse pool of model candidates and analyze the pool to find optimum models for different goal functions organized in coverage hierarchies. The selected models are finally robustly externally cross-validated by a 50 * 5-fold procedure and where applicable, externally validated,

During prediction generation, the models with the highest reliability of positive and negative predictions having the query substance within AD are applied. The predictions are provided together with the accuracy of the actual models applied as well as reliability measures for each specific prediction.

Twenty data sets from the Danish (Q)SAR Database (https://qsar.food.dtu.dk) are used to demonstrate the performance of the method. The developed binary classification models are highly accurate by cross- and external validation at many coverage levels.

---

1. Bischl B, Binder M, Lang M et al., Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges, WIREs Data Mining Knowl Discov. 2023;13:e1484, DOI: 10.1002/widm.1484.

2. Nikolov NG and Wedebye EB, High performance, large chemical coverage, or both: DanishQSAR and hierarchies of post-hoc ensemble models optimized for sensitivity, specificity or balanced accuracy, SAR & QSAR Environ Res, 2025 (manuscript submitted)