Introduction
Text classification is a core task of natural language processing and has been used in many real-world applications, such as spam detection1 and opinion mining2. Transduction learning3 is a special text classification method that uses both labelled and unlabeled samples in the training process. A graph neural network (GNN) is an effective transduction learning method4,5 and is widely used in text classification applications. This method constructs a graph to model the relationship between documents. Nodes in the figure represent text units such as text or documents, while edges are constructed based on semantic similarity between nodes. Therefore, a GNN can be used to learn and classify nodes in the figure. The advantages of this method for classification are as follows: (1) The representation of each node depends not only on itself but also on its neighbors, endowing the representation of nodes certain context information; (2) During training, the model spreads the influence of supervision labels in training and test cases through graphs and edges. Even data with no labels help to represent the learning process, yielding a higher performance.
However, the use of GNN for text classification has the following problems: (1) The method based on GNN does not regard the text as a sequence but as a set of co-occurring words. In the task of text classification, the word order relationship in the sentence plays a crucial role in the final classification result; (2) The traditional GNN does not make full use of edge features. Only 0 and 1 are used between nodes to ascertain whether there is a connection, namely, the connectivity feature; however, the edge features of the graph often have rich semantic information, such as the type of connection between nodes, connection strength, etc ., and should be represented as continuous vector features rather than binary variables.
Recent studies have shown that large-scale pre-training models are effective for various natural language processing tasks, especially text classification tasks6,7. The pre-training model takes the unsupervised corpus as the training object and can learn the rich text semantics implied in the language. However, the methods used for transducing text classification tasks prior to 20204,5,13-17 did not consider the use of pre-trained models. It was not until 2021 when Lin et al . proposed BERTGCN26, which combines BERT and GCN and demonstrates the effectiveness of pre-trained models in transductive learning.