How Does Model Scale Affect Ontology Learning? A Controlled Comparison of Qwen Series and GPT

Background and Context

The rapid iteration of Large Language Models (LLMs) has established a prevailing industry intuition that larger parameter counts inherently yield superior performance in knowledge extraction and ontology construction. However, this assumption lacks rigorous empirical validation within the specific vertical of Ontology Learning (OL). While general benchmarks often correlate scale with capability, the nuanced requirements of structured knowledge acquisition—such as precise term identification and hierarchical taxonomy discovery—remain under-explored regarding how model size translates to functional utility. This study addresses this critical gap by isolating the impact of model scale, architecture type, and lineage on OL performance, challenging the notion that parameter quantity is the sole determinant of success in specialized engineering tasks.

To ensure methodological rigor, the research employs the OntoLearner retrieval-augmented generation (RAG) pipeline as a unified evaluation framework. This approach controls for external variables by subjecting all tested models to identical information retrieval contexts, embedding models, prompt templates, and decoding settings. By eliminating biases stemming from varying retrieval qualities or prompt engineering differences, the experiment isolates the intrinsic capabilities of the models themselves. The evaluation suite encompasses 13 models from the Qwen3.5 and Qwen3.6 series, including both dense and Mixture-of-Experts (MoE) architectures, alongside closed-source GPT variants. This comprehensive selection allows for a direct comparison of architectural paradigms across a wide spectrum of parameter scales, providing a robust dataset for analyzing performance differentials in technical domains.

The experimental scope focuses on four ontology benchmarks derived from biomedicine and materials science, covering key tasks such as term type identification, taxonomy discovery, and non-taxonomic relation extraction. These domains were selected for their high complexity and the necessity for precise semantic understanding. By applying standardized metrics across these specialized fields, the study quantifies subtle performance variations that generic benchmarks often obscure. This focus on domain-specific application highlights the practical stakes of model selection, moving beyond theoretical scale metrics to assess real-world efficacy in constructing accurate, structured knowledge bases for scientific and industrial applications.

Deep Analysis

The results reveal a complex, non-monotonic relationship between model scale and performance, particularly within the Qwen3.5 dense model lineage. Increasing parameter sizes primarily drives improvements in precision rather than recall, with the most significant performance gains observed in the 9B to 27B range. This suggests that beyond a certain threshold, additional parameters contribute diminishing returns in capturing broader relationships, potentially introducing noise rather than enhancing recall. Notably, the 27B dense model outperforms significantly larger sparse models in term type identification, directly contradicting the simplistic "bigger is better" heuristic. This finding indicates that for tasks requiring high accuracy in entity classification, moderate-scale dense models can offer superior signal-to-noise ratios compared to their larger, sparser counterparts.

In the realm of taxonomy discovery, Mixture-of-Experts (MoE) architectures demonstrate distinct advantages, with larger MoE models exhibiting the strongest performance among open-weight options. This highlights the structural efficiency of MoE designs in handling complex hierarchical structures, where different experts can specialize in distinct aspects of the taxonomy. However, the study identifies non-taxonomic relation extraction as a universal bottleneck across all model scales and architectures. Performance in this area remains particularly weak on materials science ontology benchmarks, suggesting that current model architectures struggle with the intricate, non-hierarchical semantic connections prevalent in scientific literature, regardless of their parameter count.

Furthermore, the analysis underscores that architectural design and model lineage exert a more profound influence on performance than nominal parameter counts. When comparing Qwen variants with GPT models of similar scale, differences in internal architecture and training lineage prove to be more significant determinants of success than the sheer number of parameters. This implies that the internal mechanisms of knowledge representation and retrieval are critical factors in OL tasks. The data suggests that optimizing for specific architectural efficiencies, such as the specialized routing in MoE models or the dense attention mechanisms in mid-sized Qwen models, yields better returns than merely scaling up parameter volumes without architectural innovation.

Industry Impact

For industrial applications, these findings provide empirical guidance that challenges the trend of blindly deploying ultra-large models for knowledge engineering tasks. The study indicates that for ontology learning, which demands high precision, models around the 27B parameter mark or specific MoE architectures may offer a more optimal balance of cost and performance. This realization can lead to significant resource savings in deployment and inference, as organizations can avoid the computational overhead of larger models that do not proportionally improve task outcomes. It encourages a shift towards more nuanced model selection strategies, where the specific requirements of precision versus recall, and the nature of the ontology structure, dictate the choice of architecture rather than defaulting to the largest available model.

The research also emphasizes the importance of reproducibility and controlled evaluation within the open-source community. By demonstrating that standardized pipelines like OntoLearner can effectively isolate model capabilities, the study advocates for the establishment of more rigorous benchmarking protocols. This push for standardization is crucial for fair comparisons across different model families and architectures. It provides a template for future evaluations that prioritize methodological consistency, ensuring that performance claims are based on comparable conditions. This shift towards rigorous, controlled testing can help the community move away from marketing-driven metrics towards a more scientific understanding of model capabilities in specialized domains.

Moreover, the identification of non-taxonomic relation extraction as a persistent challenge highlights a critical area for future development. The study suggests that current models, regardless of scale, struggle with complex semantic relationships in specialized fields like materials science. This points to a need for targeted improvements in how models handle non-hierarchical knowledge, potentially through enhanced retrieval strategies or domain-specific fine-tuning. For industries relying on these ontologies, such as pharmaceuticals or advanced materials, this underscores the necessity of investing in specialized model optimization rather than relying on general-purpose LLMs. It signals a move towards more hybrid approaches that combine LLMs with domain-specific knowledge graphs and reasoning engines to overcome current limitations.

Outlook

Looking forward, the study suggests that the future of LLM-assisted ontology engineering lies in architectural innovation and domain-specific integration rather than mere scale expansion. The evidence that architecture and lineage outweigh nominal parameter size implies that future research should focus on designing models with structures optimized for knowledge extraction tasks. This could involve further exploration of MoE variants, hybrid dense-sparse models, or novel attention mechanisms that better capture complex semantic relationships. Developers and researchers are encouraged to prioritize these structural advancements to achieve higher performance gains without the associated computational costs of scaling parameter counts.

Additionally, the persistent difficulty in non-taxonomic relation extraction points to the need for enhanced retrieval-augmented generation strategies. Future iterations of pipelines like OntoLearner may benefit from more sophisticated retrieval mechanisms that can better surface relevant, complex relational data for the model to process. Integrating external knowledge bases or graph structures directly into the generation process could help bridge the gap between current model capabilities and the requirements of complex ontology engineering. This hybrid approach could leverage the strengths of both LLMs and structured data systems to achieve more accurate and comprehensive knowledge extraction.

Finally, the study calls for continued collaboration between the open-source community and industrial practitioners to refine evaluation standards and share best practices. By establishing common benchmarks and sharing insights on model performance in specific domains, the community can accelerate the development of more effective tools for ontology learning. This collective effort will help drive the field towards more rational, evidence-based model selection and deployment strategies. As the technology evolves, the focus will likely shift from competing on scale to competing on efficiency, precision, and the ability to integrate seamlessly into specialized scientific and industrial workflows, ultimately leading to more robust and reliable AI-assisted knowledge engineering solutions.