About this Dataset

BETH Dataset

This dataset corresponds to the paper "BETH Dataset: Real Cybersecurity Data for Anomaly Detection Research" by Kate Highnam* (@jinxmirror13), Kai Arulkumaran* (@kaixhin), Zachary Hanif*, and Nicholas R. Jennings (@LboroVC).

This paper was published in the ICML Workshop on Uncertainty and Robustness in Deep Learning 2021 and Conference on Applied Machine Learning for Information Security (CAMLIS 2021)

THIS DATASET IS STILL BEING UPDATED

Context

When deploying machine learning (ML) models in the real world, anomalous data points and shifts in the data distribution are inevitable. From a cyber security perspective, these anomalies and dataset shifts are driven by both defensive and adversarial advancement. To withstand the cost of critical system failure, the development of robust models is therefore key to the performance, protection, and longevity of deployed defensive systems.

We present the BPF-extended tracking honeypot (BETH) dataset as the first cybersecurity dataset for uncertainty and robustness benchmarking. Collected using a novel honeypot tracking system, our dataset has the following properties that make it attractive for the development of robust ML methods:

At over eight million data points, this is one of the largest cyber security datasets available
It contains modern host activity and attacks
It is fully labelled
It contains highly structured but heterogeneous features
Each host contains benign activity and at most a single attack, which is ideal for behavioural analysis and other research tasks. In addition to the described dataset

Further data is currently being collected and analysed to add alternative attack vectors to the dataset.

There are several existing cyber security datasets used in ML research, including the KDD Cup 1999 Data (Hettich & Bay, 1999), the 1998 DARPA Intrusion Detection Evaluation Dataset (Labs, 1998; Lippmann et al., 2000), the ISCX IDS 2012 dataset (Shiravi et al., 2012), and NSL-KDD (Tavallaee et al., 2009), which primarily removes duplicates from the KDD Cup 1999 Data. Each includes millions of records of realistic activity for enterprise applications, with labels for attacks or benign activity. The KDD1999, NSLKDD, and ISCX datasets contain network traffic, while the DARPA1998 dataset also includes limited process calls. However, these datasets are at best almost a decade old, and are collected on in-premise servers. In contrast, BETH contains modern host activity and activity collected from cloud services, making it relevant for current real-world deployments. In addition, some datasets include artificial user activity (Shiravi et al., 2012) while BETH contains only real activity. BETH is also one of the few datasets to include both kernel-process and network logs, providing a holistic view of malicious behaviour.

Content

The BETH dataset currently represents 8,004,918 events collected over 23 honeypots, running for about five noncontiguous hours on a major cloud provider. For benchmarking and discussion, we selected the initial subset of the process logs. This subset was further divided into training, validation, and testing sets with a rough 60/20/20 split based on host, quantity of logs generated, and the activity logged—only the test set includes an attack

The dataset is composed of two sensor logs: kernel-level process calls and network traffic. The initial benchmark subset only includes process logs. Each process call consists of 14 raw features and 2 hand-crafted labels.

See the paper for more details. For details on the events recorded within the logs, see this report.

Benchmarks

Code for our benchmarks, as detailed in the paper, are available through Github at: https://github.com/jinxmirror13/BETH_Dataset_Analysis

Acknowledgements

Thank you to Dr. Arinbjörn Kolbeinsson for his assistance in analysing the data and the reviewers for their positive feedback.

Tables

Labelled 2021may Ip 10–100–1–105 Dns

@kaggle.katehighnam_beth_dataset.labelled_2021may_ip_10_100_1_105_dns

13.75 KB
269 rows
15 columns


CREATE TABLE labelled_2021may_ip_10_100_1_105_dns (
  "timestamp" VARCHAR,
  "sourceip" VARCHAR,
  "destinationip" VARCHAR,
  "dnsquery" VARCHAR,
  "dnsanswer" VARCHAR,
  "dnsanswerttl" VARCHAR,
  "dnsquerynames" VARCHAR,
  "dnsqueryclass" VARCHAR,
  "dnsquerytype" VARCHAR,
  "numberofanswers" BIGINT,
  "dnsresponsecode" BIGINT,
  "dnsopcode" BIGINT,
  "sensorid" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled 2021may Ip 10–100–1–105

@kaggle.katehighnam_beth_dataset.labelled_2021may_ip_10_100_1_105

9.17 MB
409931 rows
13 columns


CREATE TABLE labelled_2021may_ip_10_100_1_105 (
  "timestamp" DOUBLE,
  "processid" BIGINT,
  "parentprocessid" BIGINT,
  "userid" BIGINT,
  "processname" VARCHAR,
  "hostname" VARCHAR,
  "eventid" BIGINT,
  "eventname" VARCHAR,
  "argsnum" BIGINT,
  "returnvalue" BIGINT,
  "args" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled 2021may Ip 10–100–1–186 Dns

@kaggle.katehighnam_beth_dataset.labelled_2021may_ip_10_100_1_186_dns

13.75 KB
269 rows
15 columns


CREATE TABLE labelled_2021may_ip_10_100_1_186_dns (
  "timestamp" VARCHAR,
  "sourceip" VARCHAR,
  "destinationip" VARCHAR,
  "dnsquery" VARCHAR,
  "dnsanswer" VARCHAR,
  "dnsanswerttl" VARCHAR,
  "dnsquerynames" VARCHAR,
  "dnsqueryclass" VARCHAR,
  "dnsquerytype" VARCHAR,
  "numberofanswers" BIGINT,
  "dnsresponsecode" BIGINT,
  "dnsopcode" BIGINT,
  "sensorid" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled 2021may Ip 10–100–1–186

@kaggle.katehighnam_beth_dataset.labelled_2021may_ip_10_100_1_186

16.84 MB
713867 rows
13 columns


CREATE TABLE labelled_2021may_ip_10_100_1_186 (
  "timestamp" DOUBLE,
  "processid" BIGINT,
  "parentprocessid" BIGINT,
  "userid" BIGINT,
  "processname" VARCHAR,
  "hostname" VARCHAR,
  "eventid" BIGINT,
  "eventname" VARCHAR,
  "argsnum" BIGINT,
  "returnvalue" BIGINT,
  "args" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled 2021may Ip 10–100–1–26 Dns

@kaggle.katehighnam_beth_dataset.labelled_2021may_ip_10_100_1_26_dns

13.75 KB
269 rows
15 columns


CREATE TABLE labelled_2021may_ip_10_100_1_26_dns (
  "timestamp" VARCHAR,
  "sourceip" VARCHAR,
  "destinationip" VARCHAR,
  "dnsquery" VARCHAR,
  "dnsanswer" VARCHAR,
  "dnsanswerttl" VARCHAR,
  "dnsquerynames" VARCHAR,
  "dnsqueryclass" VARCHAR,
  "dnsquerytype" VARCHAR,
  "numberofanswers" BIGINT,
  "dnsresponsecode" BIGINT,
  "dnsopcode" BIGINT,
  "sensorid" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled 2021may Ip 10–100–1–26

@kaggle.katehighnam_beth_dataset.labelled_2021may_ip_10_100_1_26

8.09 MB
378424 rows
13 columns


CREATE TABLE labelled_2021may_ip_10_100_1_26 (
  "timestamp" DOUBLE,
  "processid" BIGINT,
  "parentprocessid" BIGINT,
  "userid" BIGINT,
  "processname" VARCHAR,
  "hostname" VARCHAR,
  "eventid" BIGINT,
  "eventname" VARCHAR,
  "argsnum" BIGINT,
  "returnvalue" BIGINT,
  "args" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled 2021may Ip 10–100–1–4 Dns

@kaggle.katehighnam_beth_dataset.labelled_2021may_ip_10_100_1_4_dns

13.75 KB
269 rows
15 columns


CREATE TABLE labelled_2021may_ip_10_100_1_4_dns (
  "timestamp" VARCHAR,
  "sourceip" VARCHAR,
  "destinationip" VARCHAR,
  "dnsquery" VARCHAR,
  "dnsanswer" VARCHAR,
  "dnsanswerttl" VARCHAR,
  "dnsquerynames" VARCHAR,
  "dnsqueryclass" VARCHAR,
  "dnsquerytype" VARCHAR,
  "numberofanswers" BIGINT,
  "dnsresponsecode" BIGINT,
  "dnsopcode" BIGINT,
  "sensorid" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled 2021may Ip 10–100–1–4

@kaggle.katehighnam_beth_dataset.labelled_2021may_ip_10_100_1_4

10.86 MB
485241 rows
13 columns


CREATE TABLE labelled_2021may_ip_10_100_1_4 (
  "timestamp" DOUBLE,
  "processid" BIGINT,
  "parentprocessid" BIGINT,
  "userid" BIGINT,
  "processname" VARCHAR,
  "hostname" VARCHAR,
  "eventid" BIGINT,
  "eventname" VARCHAR,
  "argsnum" BIGINT,
  "returnvalue" BIGINT,
  "args" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled 2021may Ip 10–100–1–95 Dns

@kaggle.katehighnam_beth_dataset.labelled_2021may_ip_10_100_1_95_dns

13.75 KB
269 rows
15 columns


CREATE TABLE labelled_2021may_ip_10_100_1_95_dns (
  "timestamp" VARCHAR,
  "sourceip" VARCHAR,
  "destinationip" VARCHAR,
  "dnsquery" VARCHAR,
  "dnsanswer" VARCHAR,
  "dnsanswerttl" VARCHAR,
  "dnsquerynames" VARCHAR,
  "dnsqueryclass" VARCHAR,
  "dnsquerytype" VARCHAR,
  "numberofanswers" BIGINT,
  "dnsresponsecode" BIGINT,
  "dnsopcode" BIGINT,
  "sensorid" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled 2021may Ip 10–100–1–95

@kaggle.katehighnam_beth_dataset.labelled_2021may_ip_10_100_1_95

10.67 MB
479433 rows
13 columns


CREATE TABLE labelled_2021may_ip_10_100_1_95 (
  "timestamp" DOUBLE,
  "processid" BIGINT,
  "parentprocessid" BIGINT,
  "userid" BIGINT,
  "processname" VARCHAR,
  "hostname" VARCHAR,
  "eventid" BIGINT,
  "eventname" VARCHAR,
  "argsnum" BIGINT,
  "returnvalue" BIGINT,
  "args" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled 2021may Ubuntu Dns

@kaggle.katehighnam_beth_dataset.labelled_2021may_ubuntu_dns

13.75 KB
269 rows
15 columns


CREATE TABLE labelled_2021may_ubuntu_dns (
  "timestamp" VARCHAR,
  "sourceip" VARCHAR,
  "destinationip" VARCHAR,
  "dnsquery" VARCHAR,
  "dnsanswer" VARCHAR,
  "dnsanswerttl" VARCHAR,
  "dnsquerynames" VARCHAR,
  "dnsqueryclass" VARCHAR,
  "dnsquerytype" VARCHAR,
  "numberofanswers" BIGINT,
  "dnsresponsecode" BIGINT,
  "dnsopcode" BIGINT,
  "sensorid" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled 2021may Ubuntu

@kaggle.katehighnam_beth_dataset.labelled_2021may_ubuntu

2.4 MB
199222 rows
13 columns


CREATE TABLE labelled_2021may_ubuntu (
  "timestamp" DOUBLE,
  "processid" BIGINT,
  "parentprocessid" BIGINT,
  "userid" BIGINT,
  "processname" VARCHAR,
  "hostname" VARCHAR,
  "eventid" BIGINT,
  "eventname" VARCHAR,
  "argsnum" BIGINT,
  "returnvalue" BIGINT,
  "args" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled Testing Data

@kaggle.katehighnam_beth_dataset.labelled_testing_data

4.45 MB
188967 rows
16 columns


CREATE TABLE labelled_testing_data (
  "timestamp" DOUBLE,
  "processid" BIGINT,
  "threadid" BIGINT,
  "parentprocessid" BIGINT,
  "userid" BIGINT,
  "mountnamespace" BIGINT,
  "processname" VARCHAR,
  "hostname" VARCHAR,
  "eventid" BIGINT,
  "eventname" VARCHAR,
  "stackaddresses" VARCHAR,
  "argsnum" BIGINT,
  "returnvalue" BIGINT,
  "args" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled Training Data

@kaggle.katehighnam_beth_dataset.labelled_training_data

17.52 MB
763144 rows
16 columns


CREATE TABLE labelled_training_data (
  "timestamp" DOUBLE,
  "processid" BIGINT,
  "threadid" BIGINT,
  "parentprocessid" BIGINT,
  "userid" BIGINT,
  "mountnamespace" BIGINT,
  "processname" VARCHAR,
  "hostname" VARCHAR,
  "eventid" BIGINT,
  "eventname" VARCHAR,
  "stackaddresses" VARCHAR,
  "argsnum" BIGINT,
  "returnvalue" BIGINT,
  "args" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);

Labelled Validation Data

@kaggle.katehighnam_beth_dataset.labelled_validation_data

3.72 MB
188967 rows
16 columns


CREATE TABLE labelled_validation_data (
  "timestamp" DOUBLE,
  "processid" BIGINT,
  "threadid" BIGINT,
  "parentprocessid" BIGINT,
  "userid" BIGINT,
  "mountnamespace" BIGINT,
  "processname" VARCHAR,
  "hostname" VARCHAR,
  "eventid" BIGINT,
  "eventname" VARCHAR,
  "stackaddresses" VARCHAR,
  "argsnum" BIGINT,
  "returnvalue" BIGINT,
  "args" VARCHAR,
  "sus" BIGINT,
  "evil" BIGINT
);