Which Modern AI Methods Provide Accurate Predictions of Toxicological Endpoints? Analysis of Tox24 Challenge Results

The Tox24 challenge1 was designed to evaluate the progress that has been made in computational method development for the prediction of in vitro activity since the Tox21 challenge, which was organised by National Institutes of Health  National Center for Advancing Translational Sciences (NCATS).2 In this challenge, participants were tasked with developing models to predict chemical binding to transthyretin (TTR), a serum binding protein, based on chemical structure. The chemicals were tested for activity against TTR3 by the US EPA. In total 78 teams representing 27 countries participated in the Challenge which was run from May to August 2024,  and the winners were announced during the ICANN2024 conference in Lugano.

 Eleven models had RMSE non-significantly different to the top model. The winner #1, as well as runners-up #2 and #6, developed their models using the OCHEM (https://ochem.eu) platform.4 This impressive result further confirms the high accuracy of OCHEM, which was previously used to develop the winning models for several other challenges. In 8 out of 11 models, the authors used at least one representation learning method. They included Graph Neural Networks, as well as Natural Language Processing (NLP) methods based on SMILES data processing or Foundation Chemistry Models. Among traditional methods, the most popular were decision trees but other traditional methods based on fully connected deep neural networks and SVM were also used. The majority of the descriptor-based models were developed using 2D descriptors and only one team used 3D descriptors. The use of mixtures, tautomers and other bioactivity data were also techniques which contributed to the winning strategies. 

Many of the approaches used by top-ranked models are less than five years old and did not exist during the Tox21 Challenge. These observations clearly demonstrate the high impact that advanced ML/AI methods have made on the field. Considering the high accuracy of novel methods, the OECD principles on the Validation of (Quantitative) Structure-Activity Relationship [(Q)SAR] Models5 may need to be extended to describe/exemplify how these approaches and their intrinsic features (e.g., consensus, multitasking, pre-training, fine-tuning, transfer learning, etc.) can be reliably used in regulatory assessments. This would allow better hazard and risk assessment of chemical compounds, which is in particular important to develop safe by design chemicals.

Acknowledgements: The challenge was co-organised by Marie Sklodowska-Curie Innovative Training Network European Industrial Doctorate grant agreement No. 956832 “Advanced machine learning for Innovative Drug Discovery” (AIDD), Horizon Europe Marie Skłodowska-Curie Actions Doctoral Network grant agreement No. 101120466 “Explainable AI for Molecules” (AiChemist) as well as by Chemical Research in Toxicology journal and ICANN2024. 

(1)    Eytcheson, S. A.; Tetko, I. V. Which Modern AI Methods Provide Accurate Predictions of Toxicological Endpoints? Analysis of Tox24 Challenge Results. ChemRxiv January 10, 2025. https://doi.org/10.26434/chemrxiv-2025-7k7x3.

(2)    Huang, R.; Xia, M.; Nguyen, D.-T.; Zhao, T.; Sakamuru, S.; Zhao, J.; Shahane, S. A.; Rossoshek, A.; Simeonov, A. Tox21Challenge to Build Predictive Models of Nuclear Receptor and Stress Response Pathways as Mediated by Exposure to Environmental Chemicals and Drugs. Front. Environ. Sci. 2016, 3. https://doi.org/10.3389/fenvs.2015.00085.

(3)    Eytcheson, S. A.; Zosel, A. D.; Olker, J. H.; Hornung, M. W.; Degitz, S. J. Screening the ToxCast Chemical Libraries for Binding to Transthyretin. Chem. Res. Toxicol. 2024, 37 (10), 1670–1681. https://doi.org/10.1021/acs.chemrestox.4c00215.

(4)    Sushko, I.; Novotarskyi, S.; Körner, R.; Pandey, A. K.; Rupp, M.; Teetz, W.; Brandmaier, S.; Abdelaziz, A.; Prokopenko, V. V.; Tanchuk, V. Y.; Todeschini, R.; Varnek, A.; Marcou, G.; Ertl, P.; Potemkin, V.; Grishina, M.; Gasteiger, J.; Schwab, C.; Baskin, I. I.; Palyulin, V. A.; Radchenko, E. V.; Welsh, W. J.; Kholodovych, V.; Chekmarev, D.; Cherkasov, A.; Aires-de-Sousa, J.; Zhang, Q.-Y.; Bender, A.; Nigsch, F.; Patiny, L.; Williams, A.; Tkachenko, V.; Tetko, I. V. Online Chemical Modeling Environment (OCHEM): Web Platform for Data Storage, Model Development and Publishing of Chemical Information. J. Comput. Aided Mol. Des. 2011, 25 (6), 533–554. https://doi.org/10.1007/s10822-011-9440-2.

(5)    Guidance Document on the Validation of (Quantitative) Structure-Activity Relationship [(Q)SAR] Models. OECD. https://doi.org/10.1787/9789264085442-en (accessed 2024-12-10).

Speakers:  Stephanie A. Eytcheson 1,2 and Igor V. Tetko 3,4,*

1 Oak Ridge Institute for Science and Education, Oak Ridge, Tennessee 37830, United States; 2 Center for Computational Toxicology and Exposure, Great Lakes Toxicology and Ecology Division, U.S. Environmental Protection Agency, Office of Research and Development, Duluth, Minnesota 55804, United States;  3Institute of Structural Biology, Molecular Targets and Therapeutics Center, Helmholtz Munich - Deutsches Forschungszentrum Für Gesundheit Und Umwelt (GmbH), 86764 Neuherberg, Germany; 4 BIGCHEM GmbH, Valerystr. 49, 85716 Unterschleißheim, Germany