BERT-Embedded Spam Messages

Context

This dataset is an extension of the original dataset which is a set of English SMS messages tagged with being spam or ham.

The dataset was created to add the possibility to work with BERT-Embeddings. Since creating these embeddings in kaggle kernels is not feasible for memory efficiency reasons, I've created them locally and provide you the original dataset plus the embedings. So in this dataset you get the original dataset plus the embeddings for each SMS message!

Please refer to the original dataset for further clarification.

Content

The dataset contains the same information as the original dataset plus the additional DiltilBERT classification embeddings.

This results in a dataset with 5574 rows and 770 columns:

• spam -> Target column specifying if the message is spam or ham
• original_message -> The original unprocessed messages
• 0 up to 768 -> columns containing the DistilBERT classification embeddings for the message, after it being processed

Inspiration

• Can you classify spam messages using the embeddings?
• Does BERT-Embeddings work better than TF-IDF?
• What is the highest ROC-AUC you can get?
• What features can be derived from the dataset?
• What is the most common words from Spam/Ham messages?
• What are some Spam messages you can't correctly classify?

Procedure for creating the dataset

HuggingFace's DistilBERT is used from their transformers package.

Jay Allamar's tutorial was followed to encode the messages using DistilBERT.

For memory efficiency reasons all messages are first stripped from punctuation and then english stopwords are removed. Then only the first 30 tokens are kept.

As per my analysis of the original dataset it can be seen that most ham messages have around 10 words and spam messages around 29 words, without stopwords. This means that once stopwords are removed from the messages, keeping the first 30 tokens might mean some information loss but not to critical. (Acrually in my analysis it is demonstrated that encoding the messages using only the first 10 tokens after processing them is enough to have a good encoding capable of achieving 0.881 ROC-AUC with a baseline random forest.)

To better understand how the embeddings were created I encourage to check out the Github repo with the script for creating the dataset.

Acknowledgements

Jay Allamar's tutorial was followed to encode the messages using DistilBERT.

The original dataset is part of the UCI Machine Learning repository and can be found here.

UCI Machine Learning urges to if you find the original dataset useful, cite the original authors found here.

Almeida, T.A., Gómez Hidalgo, J.M., Yamakami, A. Contributions to the Study of SMS Spam Filtering: New Collection and Results. Proceedings of the 2011 ACM Symposium on Document Engineering (DOCENG'11), Mountain View, CA, USA, 2011