Skip to main content
Springer Nature Link
Log in

Graph isomorphism attention network combined with pre-trained language models: a novel approach for crystal material property prediction

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Predicting crystal material properties is a central task in AI-for-science, yet existing graph neural network methods face three limitations: (1) conventional neighbor-based graph construction causes node features to be dominated by neighbors, losing atomic uniqueness, (2) such atomic graphs only capture short-range dependencies via immediate neighbors while neglecting long-range interactions through multi-hop connections, and (3) insufficient utilization of chemical formulae, often relying on simplistic one-hot encoding without material-specific knowledge. To address these, we propose the graph isomorphism attention network (GIAT), which balances self-node and neighbor information to preserve atomic characteristics while capturing short-range dependencies. Additionally, we integrate a GraphTransformer to model long-range atomic interactions, forming a complementary framework with GIAT. Furthermore, we employ matBERT, a material science-specific language model, to encode chemical formulae, leveraging domain knowledge from both individual materials and their analogs. Experiments show that our model achieves state-of-the-art performance by synergistically combining short-range (GIAT), long-range (GraphTransformer), and chemical formula embeddings (matBERT).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Code availability

The source code of GIAT-PLM-former is available if a request email is sent to professor Liang Yang.

Notes

  1. https://github.com/lbnlp/MatBERT.

  2. https://github.com/superlouis/GATGNN.

References

  1. Xie T, Grossman JC (2018) Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys Rev Lett 120:145301. https://doi.org/10.1103/PhysRevLett.120.145301

    Article  Google Scholar 

  2. Chen C, Ye W, Zuo Y, Zheng C, Ong SP (2019) Graph networks as a universal machine learning framework for molecules and crystals. Chem Mater 31(9):3564–3572. https://doi.org/10.1021/acs.chemmater.9b01294

    Article  Google Scholar 

  3. Louis S-Y, Zhao Y, Nasiri A, Wang X, Song Y, Liu F, Hu J (2020) Graph convolutional neural networks with global attention for improved materials property prediction. Phys Chem Chem Phys 22:18141–18148. https://doi.org/10.1039/D0CP01474E

    Article  Google Scholar 

  4. Xiao J, Yang L, Wang S (2024) Graph isomorphism network for materials property prediction along with explainability analysis. Comput Mater Sci 233:112619. https://doi.org/10.1016/j.commatsci.2023.112619

    Article  Google Scholar 

  5. Choudhary K, DeCost B (2021) Atomistic line graph neural network for improved materials property predictions. NPJ Comput Mater 7(1):185. https://doi.org/10.1038/s41524-021-00650-1

    Article  Google Scholar 

  6. Jain A, Ong SP, Hautier G, Chen W, Richards WD, Dacek S, Cholia S, Gunter D, Skinner D, Ceder G, Persson KA (2013) Commentary: the materials project: a materials genome approach to accelerating materials innovation. APL Mater 1(1):011002. https://doi.org/10.1063/1.4812323 (https://pubs.aip.org/aip/apm/article-pdf/doi/10.1063/1.4812323/13163869/011002_1_online.pdf)

    Article  Google Scholar 

  7. Choudhary K, Kalish I, Beams R, Tavazza F (2017) High-throughput identification and characterization of two-dimensional materials using density functional theory. Sci Rep 7(1):5179

    Article  Google Scholar 

  8. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: International conference on learning representations. https://openreview.net/forum?id=SJU4ayYgl

  9. Veličković P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y (2018) Graph attention networks. In: International conference on learning representations. https://openreview.net/forum?id=rJXMpikCZ

  10. Neugebauer J, Hickel T (2013) Density functional theory in materials science. WIREs Comput Mol Sci 3(5):438–448. https://doi.org/10.1002/wcms.1125 (https://wires.onlinelibrary.wiley.com/doi/pdf/10.1002/wcms.1125)

    Article  Google Scholar 

  11. Jong M, Chen W, Notestine R, Persson K, Ceder G, Jain A, Asta M, Gamst A (2016) A statistical learning framework for materials science: application to elastic moduli of k-nary inorganic polycrystalline compounds. Sci Rep 6(1):34256. https://doi.org/10.1038/srep34256

    Article  Google Scholar 

  12. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd international conference on international conference on machine learning. ICML’15, , vol 37, pp 448–456

  13. Cai D, Lam W (2020) Graph transformer for graph-to-sequence learning. Proc AAAI Conf Artif Intell 34:7464–7471

    Google Scholar 

  14. Walker N, Trewartha A, Huo H, Lee S, Cruse K, Dagdelen J, Dunn A, Persson K, Ceder G, Jain A (2021) The impact of domain-specific pre-training on named entity recognition tasks in materials science. SSRN Electron J. https://doi.org/10.2139/ssrn.3950755

    Article  Google Scholar 

  15. Reiser P, Neubert M, Eberhard A, Torresi L, Zhou C, Shao C, Metni H, Hoesel C, Schopmans H, Sommer T, Friederich P (2022) Graph neural networks for materials science and chemistry. Commun Mater 3(1):93. https://doi.org/10.1038/s43246-022-00315-6

    Article  Google Scholar 

  16. Wieder O, Kohlbacher S, Kuenemann M, Garon A, Ducrot P, Seidel T, Langer T (2020) A compact review of molecular property prediction with graph neural networks. Drug Discov Today Technol 37:1–12. https://doi.org/10.1016/j.ddtec.2020.11.009

    Article  Google Scholar 

  17. Omee SS, Louis S-Y, Fu N, Wei L, Dey S, Dong R, Li Q, Hu J (2022) Scalable deeper graph neural networks for high-performance materials property prediction. Patterns 3(5):100491. https://doi.org/10.1016/j.patter.2022.100491

    Article  Google Scholar 

  18. Devlin J, Chang M-W, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein J, Doran C, Solorio T (eds) Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, vol 1 (long and short papers). Association for Computational Linguistics, Minneapolis, pp 4171–4186. https://doi.org/10.18653/v1/N19-1423

  19. Acheampong FA, Nunoo-Mensah H, Chen W (2021) Transformer models for text-based emotion detection: a review of BERT-based approaches. Artif Intell Rev 54(8):5789–5829. https://doi.org/10.1007/s10462-021-09958-2

    Article  Google Scholar 

  20. Qin X, Wu Z, Zhang T, Li Y, Luan J, Wang B, Wang L, Cui J (2023) Bert-erc: fine-tuning bert is enough for emotion recognition in conversation. In: Proceedings of the thirty-seventh AAAI conference on artificial intelligence and thirty-fifth conference on innovative applications of artificial intelligence and thirteenth symposium on educational advances in artificial intelligence. AAAI’23/IAAI’23/EAAI’23. AAAI Press. https://doi.org/10.1609/aaai.v37i11.26582

  21. Kumar P, Raman B (2022) A bert based dual-channel explainable text emotion recognition system. Neural Netw 150:392–407. https://doi.org/10.1016/j.neunet.2022.03.017

    Article  Google Scholar 

  22. Wang Z, Ma J, Hu R, Luo X (2023) Predicting lattice thermal conductivity of semiconductors from atomic-information-enhanced cgcnn combined with transfer learning. Appl Phys Lett 122(15):152106. https://doi.org/10.1063/5.0142150 (https://pubs.aip.org/aip/apl/article-pdf/doi/10.1063/5.0142150/16821009/152106_1_5.0142150.pdf)

    Article  Google Scholar 

  23. Lu X, Xie Z, Wu X, Li M, Cai W (2022) Hydrogen storage metal-organic framework classification models based on crystal graph convolutional neural networks. Chem Eng Sci 259:117813. https://doi.org/10.1016/j.ces.2022.117813

    Article  Google Scholar 

  24. Lee J, Asahi R (2021) Transfer learning for materials informatics using crystal graph convolutional neural network. Comput Mater Sci 190:110314. https://doi.org/10.1016/j.commatsci.2021.110314

    Article  Google Scholar 

  25. Schütt KT, Kindermans P-J, Sauceda HE, Chmiela S, Tkatchenko A, Müller K-R (2017) Schnet: a continuous-filter convolutional neural network for modeling quantum interactions. In: Proceedings of the 31st international conference on neural information processing systems. NIPS’17. Curran Associates Inc., Red Hook, pp 992–1002

Download references

Funding

This work was supported by the Science and Technology projects of Yunnan Precious Metals Laboratory (Grant No. YPML-20240502102).

Author information

Authors and Affiliations

  1. Dalian University of Technology, Dalian, China

    Jiahao Kang, Liang Yang, Jingjie Zeng, Zhi Sun & Hongfei Lin

  2. Sino-Platinum Metals CO., LTD, Kunming, China

    Junpeng Li

  3. Hubei Institute of Aerospace Chemotechnology, Xiangyang, China

    Jiahao Kang

  4. Key Laboratory of Social Computing and Cognitive Intelligence, Ministry of Education, Dalian, China

    Liang Yang

Authors
  1. Jiahao Kang
    View author publications

    Search author on:PubMed Google Scholar

  2. Liang Yang
    View author publications

    Search author on:PubMed Google Scholar

  3. Jingjie Zeng
    View author publications

    Search author on:PubMed Google Scholar

  4. Zhi Sun
    View author publications

    Search author on:PubMed Google Scholar

  5. Hongfei Lin
    View author publications

    Search author on:PubMed Google Scholar

  6. Junpeng Li
    View author publications

    Search author on:PubMed Google Scholar

Corresponding authors

Correspondence to Liang Yang or Junpeng Li.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kang, J., Yang, L., Zeng, J. et al. Graph isomorphism attention network combined with pre-trained language models: a novel approach for crystal material property prediction. Neural Comput & Applic (2025). https://doi.org/10.1007/s00521-025-11532-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00521-025-11532-8

Keywords