基于BERT的汽车生产设备故障领域命名实体识别

doi:10.16180/j.cnki.issn1007-7820.2023.11.006

摘要/Abstract

摘要：

在汽车生产设备故障领域,中文命名实体识别时实体类别复杂,且传统词向量无法解决一词多义等问题。针对上述问题,文中提出一种基于BERT(Bidirectional Encoder Representations From Transformer)的汽车生产设备故障领域命名实体识别模型。首先,通过BERT预训练模型提取语义信息和句法特征,生成动态词向量。然后,将词向量输入到双向长短期记忆进行双向编码,获得长序列语义特征。最后,通过条件随机场进行序列解码,学习标签之间的依赖关系,得到最优的标签序列。在自建真实汽车生产设备故障领域数据集上进行实验,得到新方法的准确率、召回率和F1值分别为87.9%、89.6%和88.7%。

关键词: 设备故障, 自然语言处理, 序列标注, 命名实体识别, 预训练模型, LSTM, 条件随机场, 深度学习

Abstract:

In the field of automobile production equipment fault, the entity category of Chinese named entity is complicated, and the traditional word vector can not solve the polysemy of one word. In view of these problems, this study proposes a named entity recognition model in the field of automobile production equipment fault based on BERT(Bidirectional Encoder Representations From Transformer). First, the semantic information and syntactic features are extracted by BERT pretraining model to generate dynamic word vectors. Then, the word vector is input into bidirectional long-short term memory for bidirectional encoding to obtain the semantic features of long sequences. Finally, the conditional random field is used for sequence decoding to learn the dependency relationship between labels and obtain the optimal label sequence. Experiments are carried out on the self-built real automobile production equipment fault data set, and the accuracy, recall rate and F1 value are 87.9 %, 89.6 % and 88.7 %, respectively.

Key words: equipment fault, natural language processing, sequence labeling, named entity recognition, pre-training model, LSTM, conditional random fields, deep learning

中图分类号:

TP391

倪骥,王宇嘉,赵博. 基于BERT的汽车生产设备故障领域命名实体识别[J]. 电子科技, 2023, 36(11): 35-40.

NI Ji,WANG Yujia,ZHAO Bo. Named Entity Recognition of Automobile Production Equipment Fault Domain Based on BERT[J]. Electronic Science and Technology, 2023, 36(11): 35-40.

图/表 10

图1

图2

图3

图4

图5

表1

表2

表3

表4

表5

参考文献 22

[1]	焦凯楠, 李欣, 朱容辰. 中文领域命名实体识别综述[J]. 计算机工程与应用, 2021, 57(16):1-15. doi: 10.3778/j.issn.1002-8331.2103-0127
	Jiao Kainan, Li Xin, Zhu Rongchen. Overview of Chinese domain named entity recognition[J]. Computer Engineering and Applications, 2021, 57(16):1-15. doi: 10.3778/j.issn.1002-8331.2103-0127
[2]	Collobert R, Weston J, Bottou L, et al. Natural language processing (almost) from scratch[J]. Journal of Machine Learning Research, 2011, 12(1):2493-2537.
[3]	Lample G, Ballesteros M, Subramanian S, et al. Neural architectures for named entity recognition[C]. San Diego:The fifteenth Annual Comference of the North American Chapter of the Association for Computational Linguistics: human Language Technologys, 2016:260-270.
[4]	孟昕. 基于深度学习的法律文书识别方法研究[J]. 电子科技, 2019, 32(12):84-86.
	Meng Xin. Research on recognition method of legal documents based on deep learning[J]. Electronic Science and Technology, 2019, 32(12):84-86.
[5]	Peters M E, Neumann M, Iyyer M, et al. Deep contextualized word representations[C]. New Orleans:Proceedings of the Sixteenth Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018:2227-2237.
[6]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]. Long Beach: Conference and Workshop on Neural Information Processing Systems, 2017:5998-6008.
[7]	Devlin J, Chang M, Lee K, et al. BERT:Pre-training of deep bidirectional transformers for language understanding[C]. Minneapolis: The Seventeenth Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019:4171-4186.
[8]	李妮, 关焕梅, 杨飘, 等. 基于BERT-IDCNN-CRF的中文命名实体识别方法[J]. 山东大学学报(理学版), 2020, 55(1):102-109.
	Li Ni, Guan Huanmei, Yang Piao, et al. BERT-IDCNN-CRF for named entity recognition in Chinese[J]. Journal of Shandong University(Natural Science), 2020, 55(1):102-109.
[9]	Liu W, Fu X Y, Zhang Y, et al. Lexicon enhanced Chinese sequence labeling using BERT adapter[C]. Bangkok: Proceedings of the Fifty-ninth Annual Meeting of the Association for Computational Linguistics and the Eleventh International Joint Conference on Natural Language Processing, 2021:5847-5858.
[10]	Nie Y, Tian Y, Song Y, et al. Improving named entity recognition with attentive ensemble of syntactic information[C]. Punta Cana:Findings of the Association for Computational Linguistics: EMNLP, 2020:2782-2794.
[11]	Li J, Sun A X, Han J L, et al. A survey on deep learning for named entity recognition[J]. IEEE Transactions on Knowledge and Data Engineering, 2020, 34(1):50-70. doi: 10.1109/TKDE.2020.2981314
[12]	Qiu X P, Sun T X, Xu Y G, et al. Pre-trained models for natural language processing:A survey[J]. Science China Technological Sciences, 2020, 63(10):1871-1897. doi: 10.1007/s11431-020-1716-5
[13]	Chen J X, Huang Y J, Yang F, et al. A novel named entity recognition approach of judicial case texts based on BiLSTM-CRF[C]. Dali: The Twelfth International Conference on Advanced Computational Intelligence, 2020:263-268.
[14]	Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 1997, 9(8):1735-1780. doi: 10.1162/neco.1997.9.8.1735 pmid: 9377276
[15]	Gunawan W, Suhartono D, Purnomo F, et al. Named-entity recognition for Indonesian language using bidirectional LSTM-CNNs[J]. Procedia Computer Science, 2018, 135(22):425-432. doi: 10.1016/j.procs.2018.08.193
[16]	Torii M, Hu Z Z, Cathy H, et al. Research paper:Bio tagger[J]. Journal of the American Medical Informatics Association, 2009, 16(2):247-255. doi: 10.1197/jamia.M2844
[17]	杨雯迪, 任春华, 孙洁香. 支持汽车故障数据增值的词汇增强实体识别[J]. 现代计算机, 2021, 27(26):8-14.
	Yang Wendi, Ren Chunhua, Sun Jiexiang. Vocabulary-enhanced entity recognition that supports the value-added of automobile fault data[J]. Modern Computer, 2021, 27(26):8-14.
[18]	Yang J, Zhang Y, Li L W, et al. YEDDA:A lightweight collaborative text span annotation tool[C]. Melbourne: Proceedings of the Fifty-sixth Annual Meeting of the Association for Computational Linguistics, 2018:31-36.
[19]	Zhang C S, Zhang Y, Shi X J, et al. On incremental learning for gradient boosting decision trees[J]. Neural Processing Letters, 2019, 50(1):957-987. doi: 10.1007/s11063-019-09999-3
[20]	Zhou M X, Liu J, et al. A two-phase multi-objective evolutionary algorithm for enhancing the robustness of scale-free networks against multiple malicious attacks[J]. IEEE Transactions on Cybernetics, 2017, 47(2):539-552.
[21]	Liu Y, Lu J H, Yang J, et al. Sentiment analysis for ecommerce product reviews by deep learning model of Bert-BiGRU-Softmax[J]. Mathematical Biosciences and Engineering, 2020, 17(6):7819-7837. doi: 10.3934/mbe.2020398 pmid: 33378922
[22]	Yu B H, Wei J X. IDCNN-CRF-Based domain named entity recognition method[C]. Dalian: Proceedings of the IEEE Second International Conference on Civil Aviation Safety and Information Technology, 2020:542-546.

	N(Negative)	P(Positive)
F(False)	FN	FP
T(True)	TN	TP

实验环境	设置
操作系统	Windows
CPU	Inter Core i7-10700K
Python	3.6
TensorFlow	1.14.0

参数	含义	数值
optimizer	优化器	Adam
learning_rate	学习率	0.001
dropout	丢失率	0.5
batch_size	批次大小	32
embedding_dim	词嵌入维数	300
lstm_dim	隐藏层维度	200

模型	P	R	F1	Speed (s/step)
IDCNN-CRF	76.9	65.2	70.6	3.50
BIGRU-CRF	77.2	77.3	77.2	4.74
BiLSTM-CRF	84.2	84.9	84.5	19.22
BERT-BiLSTM-CRF	87.9	89.6	88.7	430.00

实体	模型	P	R	F1
设备名称	IDCNN-CRF	70.6	73.8	72.2
	BIGRU-CRF	74.7	79.5	77.0
	BILSTM-CRF	77.6	83.9	80.6
	BERT-BILSTM-CRF	80.2	85.7	82.9
故障类型	IDCNN-CRF	40.2	52.1	45.4
	BIGRU-CRF	44.6	54.2	48.9
	BILSTM-CRF	48.3	56.2	52.0
	BERT-BILSTM-CRF	65.2	58.0	61.4
属性信息	IDCNN-CRF	39.6	32.5	35.7
	BIGRU-CRF	40.2	44.6	42.3
	BILSTM-CRF	42.1	48.2	44.9
	BERT-BILSTM-CRF	62.5	63.3	62.9
操作动作	IDCNN-CRF	80.3	82.5	81.4
	BIGRU-CRF	87.1	89.7	88.4
	BILSTM-CRF	93.9	94.9	94.4
	BERT-BILSTM-CRF	96.6	95.9	96.2
正常状态	IDCNN-CRF	83.1	80.6	81.8
	BIGRU-CRF	86.7	85.3	86.0
	BILSTM-CRF	89.6	85.6	87.6
	BERT-BILSTM-CRF	90.2	89.8	90.0
异常状态	IDCNN-CRF	62.4	50.2	55.6
	BIGRU-CRF	68.5	65.9	67.2
	BILSTM-CRF	71.6	59.5	65.0
	BERT-BILSTM-CRF	78.4	82.0	80.2