AutoMM for Chinese Named Entity Recognition#
In this tutorial, we will demonstrate how to use AutoMM for Chinese Named Entity Recognition using an e-commerce dataset extracted from one of the most popular online marketplaces, TaoBao.com. The dataset is collected and labelled by Jie et al. and the text column mainly consists of product descriptions. The following figure shows an example of Taobao product description.
Load the Data#
We have preprocessed the dataset to make it ready-to-use with AutoMM.
import autogluon.multimodal
from autogluon.core.utils.loaders import load_pd
from autogluon.multimodal.utils import visualize_ner
train_data = load_pd.load('https://automl-mm-bench.s3.amazonaws.com/ner/taobao-ner/chinese_ner_train.csv')
dev_data = load_pd.load('https://automl-mm-bench.s3.amazonaws.com/ner/taobao-ner/chinese_ner_dev.csv')
train_data.head(5)
| text_snippet | entity_annotations | |
|---|---|---|
| 0 | 雄争霸点卡/七雄争霸元宝/七雄争霸100元1000元宝直充,自动充值 | [{"entity_group": "HCCX", "start": 3, "end": 5... |
| 1 | 简约韩版粗跟艾熙百思图亲子鞋冬季百搭街头母女圆头翻边绒面厚底 | [{"entity_group": "HPPX", "start": 6, "end": 8... |
| 2 | 羚跑商务背包双肩包男士防盗多功能出差韩版休闲15.6寸电脑包皮潮 | [{"entity_group": "HPPX", "start": 0, "end": 2... |
| 3 | 热水袋防爆充电暖宝卡通毛绒萌萌可爱注水暖宫暖手宝暖水袋 | [{"entity_group": "HCCX", "start": 0, "end": 3... |
| 4 | 童装11周岁13儿童夏装男童套装2017新款10中大童15男孩12秋季5潮7 | [{"entity_group": "HCCX", "start": 0, "end": 2... |
HPPX, HCCX, XH, and MISC stand for brand, product, pattern, and Miscellaneous information (e.g., product Specification), respectively. Let’s visualize one of the examples, which is about online games top up services.
visualize_ner(train_data["text_snippet"].iloc[0], train_data["entity_annotations"].iloc[0])
雄争霸点卡 HCCX /七雄争霸 MISC 元宝 HCCX /七雄争霸 MISC 100元 MISC 1000 MISC 元宝 HCCX 直充,自动充值
Training#
With AutoMM, the process of Chinese entity recognition is the same as English entity recognition.
All you need to do is to select a suitable foundation model checkpoint that are pretrained on Chinese or multilingual documents.
Here we use the 'hfl/chinese-lert-small' backbone for demonstration purpose.
Now, let’s create a predictor for named entity recognition by setting the problem_type to ner and specifying the label column. Afterwards, we call predictor.fit() to train the model for a few minutes.
from autogluon.multimodal import MultiModalPredictor
import uuid
label_col = "entity_annotations"
model_path = f"./tmp/{uuid.uuid4().hex}-automm_ner" # You can rename it to the model path you like
predictor = MultiModalPredictor(problem_type="ner", label=label_col, path=model_path)
predictor.fit(
train_data=train_data,
hyperparameters={'model.ner_text.checkpoint_name':'hfl/chinese-lert-small'},
time_limit=300, #second
)
Global seed set to 0
AutoMM starts to create your model. ✨
- AutoGluon version is 0.8.2b20230630.
- Pytorch version is 1.13.1+cu117.
- Model will be saved to "/home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner".
- Validation metric is "ner_token_f1".
- To track the learning progress, you can open a terminal and launch Tensorboard:
```shell
# Assume you have installed tensorboard
tensorboard --logdir /home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner
```
Enjoy your coffee, and let AutoMM do the job ☕☕☕ Learn more at https://auto.gluon.ai
1 GPUs are detected, and 1 GPUs will be used.
- GPU 0 name: Tesla T4
- GPU 0 memory: 15.74GB/15.84GB (Free/Total)
CUDA version is 11.7.
Using 16bit None Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
--------------------------------------------------------
0 | model | HFAutoModelForNER | 15.1 M
1 | validation_metric | MulticlassF1Score | 0
2 | loss_func | CrossEntropyLoss | 0
--------------------------------------------------------
15.1 M Trainable params
0 Non-trainable params
15.1 M Total params
30.173 Total estimated model params size (MB)
Epoch 0, global step 21: 'val_ner_token_f1' reached 0.25593 (best 0.25593), saving model to '/home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner/epoch=0-step=21.ckpt' as top 3
Epoch 0, global step 42: 'val_ner_token_f1' reached 0.68003 (best 0.68003), saving model to '/home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner/epoch=0-step=42.ckpt' as top 3
Epoch 1, global step 64: 'val_ner_token_f1' reached 0.75582 (best 0.75582), saving model to '/home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner/epoch=1-step=64.ckpt' as top 3
Epoch 1, global step 85: 'val_ner_token_f1' reached 0.76757 (best 0.76757), saving model to '/home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner/epoch=1-step=85.ckpt' as top 3
Epoch 2, global step 107: 'val_ner_token_f1' reached 0.82446 (best 0.82446), saving model to '/home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner/epoch=2-step=107.ckpt' as top 3
Epoch 2, global step 128: 'val_ner_token_f1' reached 0.82700 (best 0.82700), saving model to '/home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner/epoch=2-step=128.ckpt' as top 3
Epoch 3, global step 150: 'val_ner_token_f1' reached 0.82654 (best 0.82700), saving model to '/home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner/epoch=3-step=150.ckpt' as top 3
Epoch 3, global step 171: 'val_ner_token_f1' reached 0.82608 (best 0.82700), saving model to '/home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner/epoch=3-step=171.ckpt' as top 3
Epoch 4, global step 193: 'val_ner_token_f1' reached 0.84888 (best 0.84888), saving model to '/home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner/epoch=4-step=193.ckpt' as top 3
Time limit reached. Elapsed time is 0:05:00. Signaling Trainer to stop.
Epoch 4, global step 207: 'val_ner_token_f1' reached 0.85925 (best 0.85925), saving model to '/home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner/epoch=4-step=207.ckpt' as top 3
Start to fuse 3 checkpoints via the greedy soup algorithm.
AutoMM has created your model 🎉🎉🎉
- To load the model, use the code below:
```python
from autogluon.multimodal import MultiModalPredictor
predictor = MultiModalPredictor.load("/home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner")
```
- You can open a terminal and launch Tensorboard to visualize the training log:
```shell
# Assume you have installed tensorboard
tensorboard --logdir /home/ci/autogluon/docs/tutorials/multimodal/text_prediction/tmp/ddaa3b627d7141d2bf50a9c502b05724-automm_ner
```
- If you are not satisfied with the model, try to increase the training time,
adjust the hyperparameters (https://auto.gluon.ai/stable/tutorials/multimodal/advanced_topics/customization.html),
or post issues on GitHub: https://github.com/autogluon/autogluon
<autogluon.multimodal.predictor.MultiModalPredictor at 0x7f1c5815f580>
Evaluation#
To check the model performance on the test dataset, all you need to do is to call predictor.evaluate(...).
predictor.evaluate(dev_data)
{'hccx': {'precision': 0.761750986724076,
'recall': 0.8322226577812623,
'f1': 0.7954289996253279,
'number': 2551},
'hppx': {'precision': 0.5589225589225589,
'recall': 0.5971223021582733,
'f1': 0.5773913043478262,
'number': 278},
'misc': {'precision': 0.5747126436781609,
'recall': 0.6944444444444444,
'f1': 0.6289308176100629,
'number': 504},
'xh': {'precision': 0.6360153256704981,
'recall': 0.6974789915966386,
'f1': 0.6653306613226453,
'number': 238},
'overall_precision': 0.7094081942336874,
'overall_recall': 0.7854942593111174,
'overall_f1': 0.7455149501661129,
'overall_accuracy': 0.8667538006466259}
Prediction and Visualization#
You can easily obtain the predictions given an input sentence by by calling predictor.predict(...).
output = predictor.predict(dev_data)
visualize_ner(dev_data["text_snippet"].iloc[0], output[0])
家用防尘厨房厨师帽子 HCCX 车间工厂鸭舌工作帽 HCCX 男女食堂餐厅食品卫生帽 HCCX
Now, let’s make predictions on the rabbit toy example.
sentence = "2023年兔年挂件新年装饰品小挂饰乔迁之喜门挂小兔子"
predictions = predictor.predict({'text_snippet': [sentence]})
visualize_ner(sentence, predictions[0])
202 HCCX 3年兔年挂件 HCCX 新年装饰品 HCCX 小挂饰 HCCX 乔 HPPX 迁 MISC 之 HPPX 喜 HCCX 门挂 HCCX 小兔子 HCCX
Other Examples#
You may go to AutoMM Examples to explore other examples about AutoMM.
Customization#
To learn how to customize AutoMM, please refer to Customize AutoMM.