A RAG Approach for Generating Competency Questions in Ontology Engineering

Competency Question (CQ) formulation is essential in ontology development but often demands significant effort from domain experts. This study introduces a Retrieval-Augmented Generation (RAG) approach that leverages Large Language Models (LLMs) to automate CQ generation using scientific papers as the domain knowledge base. The research examines how varying the number of input papers and adjusting the LLM’s temperature settings affect performance. Experiments conducted with GPT-4 across two domain ontology tasks reveal that incorporating relevant domain knowledge into the RAG framework enhances CQ generation compared to zero-shot prompting. Evaluation metrics, including precision and consistency, indicate that this method effectively reduces the reliance on manual CQ crafting by domain experts.

Recommended citation: Pan, X., Ossenbruggen, J.v., de Boer, V., Huang, Z. (2025). A RAG Approach for Generating Competency Questions in Ontology Engineering. In: Sfakakis, M., Garoufallou, E., Damigos, M., Salaba, A., Papatheodorou, C. (eds) Metadata and Semantic Research. MTSR 2024. Communications in Computer and Information Science, vol 2331. Springer, Cham. https://doi.org/10.1007/978-3-031-81974-2_6
Download Paper | Download Slides