Yuan He ❲何源❳
Logo Research Associate in Computer Science

I am currently a postdoctoral Research Associate in the Knowledge Representation and Reasoning (KRR) group at the Department of Computer Science, University of Oxford, where I also obtained my PhD (DPhil) degree. I will join Amazon Rufus as an Applied Scientist early next year.

My research interests revolve around Natural Language Processing, Knowledge Engineering, and Deep Learning, currently with the following specific topics:

  • Knowledge and Hallucinations in Large Language Models
  • Large Language Models for Knowledge Engineering
  • Geometric Learning of Language Models
  • Domain-specific Training of Large Language Models
  • Multi-Agent (Agentic) Workflows
I am also the main contributor of several packages and resources, notably: DeepOnto, OAEI Bio-ML, and HierarchyTransformers.

Google Scholar Semantic Scholar GitHub LinkedIn Twitter ORCID
News
2024
Invited Talk at Talkech by CGYS
Dec
Paper accepted at NeurIPS 2024!
Sep
Got an Applied Scientist offer from Amazon Rufus! Career
Sep
Officially became a PhD with leave to supplicate! Degree
Sep
Invited Talk at GenAI BootCamp by GetSeen Ventures.
Aug
Became a Research Associate at Oxford. Career
Apr
Paper accepted in the Semantic Web journal.
Mar
Paper accepted at ESWC 2024.
Feb
2023
Won the Best Resource Paper Runner-Up at CIKM 2023. Award
Oct
Poster paper accepted at ISWC 2023.
Sep
Two papers accepted at CIKM 2023.
Sep
Joined the Program Committee for AAAI 2024.
Aug
Joined the Program Committee for CIKM 2023.
Jun
Paper accepted in Findings of ACL 2023.
May
Paper accepted in the WWW journal.
Mar
Start teaching KRR classes.
Jan
Rebuilt DeepOnto.
Jan
2022
Nominated for Best Resource Paper candidate at ISWC 2022. Award
Oct
Started organising the Bio-ML track of OAEI.
Oct
Paper accepted at DL4KG@ISWC 2022 workshop.
Sep
Joined the Program Committee for AAAI 2023.
Aug
Joined the Program Committee for ISWC 2022.
Jul
Paper accepted at ISWC 2022.
Jul
Attended and Won the Best Research Report Award at International Semantic Web Summer School (ISWS 2022). Award
Jul
Presented paper accepted at AAAI 2022.
Feb
Started teaching Knowledge Representation & Reasoning (KRR) classes at Oxford.
Jan
2021
Paper accepted at AAAI 2022.
Dec
Passed the Transfer of Status for Oxford PhD. Degree
Dec
Presented paper accepted at OM@ISWC 2021 workshop.
Oct
Paper accepted at OM@ISWC 2021 workshop.
Aug
Started the Oxford-SRUK Ontology Alignment project.
Jan
2020
Presented paper accepted at AACL 2020.
Dec
Provided teaching support for MAT marking at Oxford.
Nov
Education
  • University of Oxford
    University of Oxford
    Oct 2020 - Sep 2024
    DPhil (PhD) in Computer Science
    Thesis: Language Models for Ontology Engineering [Link]
    Supervisor: Prof. Ian Horrocks, Prof. Bernardo Cuenca Grau, Dr Jiaoyan Chen
    Funding: Fully funded by Samsung Research UK
  • University of Edinburgh
    University of Edinburgh
    Sep 2016 - May 2020
    BSc (Hons) Artificial Intelligence and Mathematics
    Thesis: Incorporating Phonetic Information in Model Design for Machine Transliteration
    Supervisor: Dr. Shay Cohen
    Grade: First class and ranked top in the class
Experience
  • University of Oxford
    University of Oxford
    Research Associate
    Apr 2024 - now
  • Teaching Assistant (Structured Data)
    Jun 2024 - Jun 2024
  • Class Tutor (KRR)
    Jan 2022 - May 2023
  • University of Edinburgh
    University of Edinburgh
    Lab Demonstrator (Reasoning and Agents)
    Jan 2019 - May 2019
  • Research Intern (Multi-lingual Machine Transliteration)
    Jun 2018 - Aug 2018
  • Research Assistant (NLP for Finance)
    May 2017 - Dec 2019
Honors & Awards
  • Best Resource Paper Runner-Up at the ACM International Conference on Information and Knowledge Management (CIKM) [Certificate]
    2023
  • Best Resource Paper Candidate at the International Semantic Web Conference (ISWC) [Nomination]
    2022
  • Best Research Report Award from the International Semantic Web Summer School (ISWS) [Certificate]
    2022
  • PhD Scholarship from Samsung Research UK
    2021
  • The 2020 Joint Class Prize (Ranked Top 1) for the degree of BSc in Artificial Intelligence and Mathematics [Certificate]
    2020
Professional Services
  • Organiser & Chair: OAEI Bio-ML Track @ ISWC, ELMKE Workshop @EKAW
  • Program Committee Member: CIKM, AAAI, ISWC, ESWC
  • Conference Reviewer: ICLR, NeurIPS, ARR (ACL, EMNLP, NAACL, etc.), ECML PKDD
  • Journal Review: Journal of Bioinformatics, Journal of Biomedical Semantics, Journal of Web Semantics, Semantic Web Journal, Data Mining and Knowledge Discovery
Selected Publications (view all )
Language Models as Hierarchy Encoders
Language Models as Hierarchy Encoders

Yuan He, Zhangdie Yuan, Jiaoyan Chen, Ian Horrocks

NeurIPS 2024

TL;DR: We introduce a novel approach to re-train transformer encoder-based language models as Hierarchy Transformer encoders (HiTs), leveraging the expansive nature of hyperbolic psace.

Abstract: Interpreting hierarchical structures latent in language is a key limitation of current language models (LMs). While previous research has implicitly leveraged these hierarchies to enhance LMs, approaches for their explicit encoding are yet to be explored. To address this, we introduce a novel approach to re-train transformer encoder-based LMs as Hierarchy Transformer encoders (HiTs), harnessing the expansive nature of hyperbolic space. Our method situates the output embedding space of pre-trained LMs within a Poincaré ball with a curvature that adapts to the embedding dimension, followed by training on hyperbolic clustering and centripetal losses. These losses are designed to effectively cluster related entities (input as texts) and organise them hierarchically. We evaluate HiTs against pre-trained LMs, standard fine-tuned LMs, and several hyperbolic embedding baselines, focusing on their capabilities in simulating transitive inference, predicting subsumptions, and transferring knowledge across hierarchies. The results demonstrate that HiTs consistently outperform all baselines in these tasks, underscoring the effectiveness and transferability of our re-trained hierarchy encoders.

[Preprint] [Poster & Slides] [Website] [HuggingFace]

Language Models as Hierarchy Encoders
Language Models as Hierarchy Encoders

Yuan He, Zhangdie Yuan, Jiaoyan Chen, Ian Horrocks

NeurIPS 2024

TL;DR: We introduce a novel approach to re-train transformer encoder-based language models as Hierarchy Transformer encoders (HiTs), leveraging the expansive nature of hyperbolic psace.

Abstract: Interpreting hierarchical structures latent in language is a key limitation of current language models (LMs). While previous research has implicitly leveraged these hierarchies to enhance LMs, approaches for their explicit encoding are yet to be explored. To address this, we introduce a novel approach to re-train transformer encoder-based LMs as Hierarchy Transformer encoders (HiTs), harnessing the expansive nature of hyperbolic space. Our method situates the output embedding space of pre-trained LMs within a Poincaré ball with a curvature that adapts to the embedding dimension, followed by training on hyperbolic clustering and centripetal losses. These losses are designed to effectively cluster related entities (input as texts) and organise them hierarchically. We evaluate HiTs against pre-trained LMs, standard fine-tuned LMs, and several hyperbolic embedding baselines, focusing on their capabilities in simulating transitive inference, predicting subsumptions, and transferring knowledge across hierarchies. The results demonstrate that HiTs consistently outperform all baselines in these tasks, underscoring the effectiveness and transferability of our re-trained hierarchy encoders.

[Preprint] [Poster & Slides] [Website] [HuggingFace]

DeepOnto: A Package for Ontology Engineering with Deep Learning
DeepOnto: A Package for Ontology Engineering with Deep Learning

Yuan He, Jiaoyan Chen, Hang Dong, Ian Horrocks, Carlo Allocca, Taehun Kim, Brahmananda Sapkota

Semantic Web 2024

TL;DR: A Python package for ontology engineering with deep learning and language models.

Abstract: Integrating deep learning techniques, particularly language models (LMs), with knowledge representation techniques like ontologies has raised widespread attention, urging the need of a platform that supports both paradigms. Although packages such as OWL API and Jena offer robust support for basic ontology processing features, they lack the capability to transform various types of information within ontologies into formats suitable for downstream deep learning-based applications. Moreover, widely-used ontology APIs are primarily Java-based while deep learning frameworks like PyTorch and Tensorflow are mainly for Python programming. To address the needs, we present DeepOnto, a Python package designed for ontology engineering with deep learning. The package encompasses a core ontology processing module founded on the widely-recognised and reliable OWL API, encapsulating its fundamental features in a more "Pythonic" manner and extending its capabilities to incorporate other essential components including reasoning, verbalisation, normalisation, taxonomy, projection, and more. Building on this module, DeepOnto offers a suite of tools, resources, and algorithms that support various ontology engineering tasks, such as ontology alignment and completion, by harnessing deep learning methods, primarily pre-trained LMs. In this paper, we also demonstrate the practical utility of DeepOnto through two use-cases: the Digital Health Coaching in Samsung Research UK and the Bio-ML track of the Ontology Alignment Evaluation Initiative (OAEI).

[Preprint] [Paper] [Website]

DeepOnto: A Package for Ontology Engineering with Deep Learning
DeepOnto: A Package for Ontology Engineering with Deep Learning

Yuan He, Jiaoyan Chen, Hang Dong, Ian Horrocks, Carlo Allocca, Taehun Kim, Brahmananda Sapkota

Semantic Web 2024

TL;DR: A Python package for ontology engineering with deep learning and language models.

Abstract: Integrating deep learning techniques, particularly language models (LMs), with knowledge representation techniques like ontologies has raised widespread attention, urging the need of a platform that supports both paradigms. Although packages such as OWL API and Jena offer robust support for basic ontology processing features, they lack the capability to transform various types of information within ontologies into formats suitable for downstream deep learning-based applications. Moreover, widely-used ontology APIs are primarily Java-based while deep learning frameworks like PyTorch and Tensorflow are mainly for Python programming. To address the needs, we present DeepOnto, a Python package designed for ontology engineering with deep learning. The package encompasses a core ontology processing module founded on the widely-recognised and reliable OWL API, encapsulating its fundamental features in a more "Pythonic" manner and extending its capabilities to incorporate other essential components including reasoning, verbalisation, normalisation, taxonomy, projection, and more. Building on this module, DeepOnto offers a suite of tools, resources, and algorithms that support various ontology engineering tasks, such as ontology alignment and completion, by harnessing deep learning methods, primarily pre-trained LMs. In this paper, we also demonstrate the practical utility of DeepOnto through two use-cases: the Digital Health Coaching in Samsung Research UK and the Bio-ML track of the Ontology Alignment Evaluation Initiative (OAEI).

[Preprint] [Paper] [Website]

Language Model Analysis for Ontology Subsumption Inference
Language Model Analysis for Ontology Subsumption Inference

Yuan He, Jiaoyan Chen, Ernesto Jiménez-Ruiz, Hang Dong, Ian Horrocks

ACL (Findings) 2023

TL;DR: Probing the conceptual (ontological) knowledge in pre-trained language models.

Abstract: Investigating whether pre-trained language models (LMs) can function as knowledge bases (KBs) has raised wide research interests recently. However, existing works focus on simple, triple-based, relational KBs, but omit more sophisticated, logic-based, conceptualised KBs such as OWL ontologies. To investigate an LM's knowledge of ontologies, we propose OntoLAMA, a set of inference-based probing tasks and datasets from ontology subsumption axioms involving both atomic and complex concepts. We conduct extensive experiments on ontologies of different domains and scales, and our results demonstrate that LMs encode relatively less background knowledge of Subsumption Inference (SI) than traditional Natural Language Inference (NLI) but can improve on SI significantly when a small number of samples are given. We will open-source our code and datasets.

[Preprint] [Paper] [Poster]

Language Model Analysis for Ontology Subsumption Inference
Language Model Analysis for Ontology Subsumption Inference

Yuan He, Jiaoyan Chen, Ernesto Jiménez-Ruiz, Hang Dong, Ian Horrocks

ACL (Findings) 2023

TL;DR: Probing the conceptual (ontological) knowledge in pre-trained language models.

Abstract: Investigating whether pre-trained language models (LMs) can function as knowledge bases (KBs) has raised wide research interests recently. However, existing works focus on simple, triple-based, relational KBs, but omit more sophisticated, logic-based, conceptualised KBs such as OWL ontologies. To investigate an LM's knowledge of ontologies, we propose OntoLAMA, a set of inference-based probing tasks and datasets from ontology subsumption axioms involving both atomic and complex concepts. We conduct extensive experiments on ontologies of different domains and scales, and our results demonstrate that LMs encode relatively less background knowledge of Subsumption Inference (SI) than traditional Natural Language Inference (NLI) but can improve on SI significantly when a small number of samples are given. We will open-source our code and datasets.

[Preprint] [Paper] [Poster]

Contextual Semantic Embeddings for Ontology Subsumption Prediction
Contextual Semantic Embeddings for Ontology Subsumption Prediction

Jiaoyan Chen, Yuan He, Ernesto Jiménez-Ruiz, Hang Dong, Ian Horrocks

World Wide Web 2023

TL;DR: Fine-tuning BERT for ontology subsumption prediction.

Abstract: Automating ontology construction and curation is an important but challenging task in knowledge engineering and artificial intelligence. Prediction by machine learning techniques such as contextual semantic embedding is a promising direction, but the relevant research is still preliminary especially for expressive ontologies in Web Ontology Language (OWL). In this paper, we present a new subsumption prediction method named BERTSubs for classes of OWL ontology. It exploits the pre-trained language model BERT to compute contextual embeddings of a class, where customized templates are proposed to incorporate the class context (e.g., neighbouring classes) and the logical existential restriction. BERTSubs is able to predict multiple kinds of subsumers including named classes from the same ontology or another ontology, and existential restrictions from the same ontology. Extensive evaluation on five real-world ontologies for three different subsumption tasks has shown the effectiveness of the templates and that BERTSubs can dramatically outperform the baselines that use (literal-aware) knowledge graph embeddings, non-contextual word embeddings and the state-of-the-art OWL ontology embeddings.

[Preprint] [Paper]

Contextual Semantic Embeddings for Ontology Subsumption Prediction
Contextual Semantic Embeddings for Ontology Subsumption Prediction

Jiaoyan Chen, Yuan He, Ernesto Jiménez-Ruiz, Hang Dong, Ian Horrocks

World Wide Web 2023

TL;DR: Fine-tuning BERT for ontology subsumption prediction.

Abstract: Automating ontology construction and curation is an important but challenging task in knowledge engineering and artificial intelligence. Prediction by machine learning techniques such as contextual semantic embedding is a promising direction, but the relevant research is still preliminary especially for expressive ontologies in Web Ontology Language (OWL). In this paper, we present a new subsumption prediction method named BERTSubs for classes of OWL ontology. It exploits the pre-trained language model BERT to compute contextual embeddings of a class, where customized templates are proposed to incorporate the class context (e.g., neighbouring classes) and the logical existential restriction. BERTSubs is able to predict multiple kinds of subsumers including named classes from the same ontology or another ontology, and existential restrictions from the same ontology. Extensive evaluation on five real-world ontologies for three different subsumption tasks has shown the effectiveness of the templates and that BERTSubs can dramatically outperform the baselines that use (literal-aware) knowledge graph embeddings, non-contextual word embeddings and the state-of-the-art OWL ontology embeddings.

[Preprint] [Paper]

BERTMap: A BERT-based Ontology Alignment System
BERTMap: A BERT-based Ontology Alignment System

Yuan He, Jiaoyan Chen, Denvar Antonyrajah, Ian Horrocks

AAAI 2022

TL;DR: We introduce BERTMap, a pipeline ontology alignment system that leverages textual information from input ontologies to fine-tune BERT for lexical matching, structural and logical information to further refine the output mappings.

Abstract: Ontology alignment (a.k.a ontology matching (OM)) plays a critical role in knowledge integration. Owing to the success of machine learning in many domains, it has been applied in OM. However, the existing methods, which often adopt ad-hoc feature engineering or non-contextual word embeddings, have not yet outperformed rule-based systems especially in an unsupervised setting. In this paper, we propose a novel OM system named BERTMap which can support both unsupervised and semi-supervised settings. It first predicts mappings using a classifier based on fine-tuning the contextual embedding model BERT on text semantics corpora extracted from ontologies, and then refines the mappings through extension and repair by utilizing the ontology structure and logic. Our evaluation with three alignment tasks on biomedical ontologies demonstrates that BERTMap can often perform better than the leading OM systems LogMap and AML.

[Preprint] [Paper] [Slides] [Poster]

BERTMap: A BERT-based Ontology Alignment System
BERTMap: A BERT-based Ontology Alignment System

Yuan He, Jiaoyan Chen, Denvar Antonyrajah, Ian Horrocks

AAAI 2022

TL;DR: We introduce BERTMap, a pipeline ontology alignment system that leverages textual information from input ontologies to fine-tune BERT for lexical matching, structural and logical information to further refine the output mappings.

Abstract: Ontology alignment (a.k.a ontology matching (OM)) plays a critical role in knowledge integration. Owing to the success of machine learning in many domains, it has been applied in OM. However, the existing methods, which often adopt ad-hoc feature engineering or non-contextual word embeddings, have not yet outperformed rule-based systems especially in an unsupervised setting. In this paper, we propose a novel OM system named BERTMap which can support both unsupervised and semi-supervised settings. It first predicts mappings using a classifier based on fine-tuning the contextual embedding model BERT on text semantics corpora extracted from ontologies, and then refines the mappings through extension and repair by utilizing the ontology structure and logic. Our evaluation with three alignment tasks on biomedical ontologies demonstrates that BERTMap can often perform better than the leading OM systems LogMap and AML.

[Preprint] [Paper] [Slides] [Poster]

All publications
Amateur
  • Literature: I am an amateur writer of novels, poetry, and prose specific to Chinese literature from a very young age.
  • Music: I am an amateur music composer (took a university-level class for composition), singer, and pianist (with an amateur level eight certificate).
  • Sport: I was a member of the university badminton squad (OuBaC) at Oxford [Photo]
Learn more