o:8173 Scientific claim verification with fine-tuned NLI models en This paper introduces the foundation for the third component of a pioneering open-source scientific questionanswering system. The system is designed to provide referenced, automatically vetted, and verifiable answers in the scientific domain where hallucinations and misinformation are intolerable. This Verification Engine is based on models fine-tuned for the Natural Language Inference task using an additionally processed SciFact dataset. Our experiments, involving eight fine-tuned models based on RoBERTa Large, XLM RoBERTa Large, DeBERTa, and DeBERTa SQuAD, show promising results. Notably, the DeBERTa model fine-tuned on our dataset achieved the highest F1 score of 88%. Furthermore, evaluating our best model on the HealthVer dataset resulted in an F1 score of 48%, outperforming other models by more than 12%. Additionally, our model demonstrated superior performance with a 7% absolute increase in F1 score compared to the bestperforming GPT-4 model on the same test set in a zero-shot regime. These findings suggest that our system can significantly enhance scientists’ productivity while fostering trust in the use of generative language models in scientific environments. claim verification, deep learning models, natural language inference, PubMed, SciFact dataset 1552100 978-989-758-716-0 1552101 2184-3228 1552099 10.5220/0012900000003838 2025-08-28T08:32:25.134Z 44 yes 46 Miloš Košprdić Istraživačko-razvojni institut za veštačku inteligenciju Srbije 0000-0001-6902-3639 46 Adela Ljajić Istraživačko-razvojni institut za veštačku inteligenciju Srbije 0000-0001-7326-059X 46 Darija Medvecki Istraživačko-razvojni institut za veštačku inteligenciju Srbije 0000-0002-4180-0050 46 Bojana Bašaragin Istraživačko-razvojni institut za veštačku inteligenciju Srbije 0000-0002-7679-1676 46 Nikola Milošević Istraživačko-razvojni institut za veštačku inteligenciju Srbije 0000-0003-2706-9676 application/pdf 305507 https://unilib.phaidrabg.rs/o:8173 no yes 18 70 92000004 71A08 Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - KMIS 3 15 25 SCITEPRESS – Science and Technology Publications, Lda 2024