Probelab Network Crawls Dataset

Nebula is Probelab's DHT crawler and monitor that is designed to track the liveliness and availability of peers. Nebula-based experiments are aimed at monitoring and improving the resilience and reliability of distributed systems by developing better tools for monitoring and managing decentralized peer-to-peer networks.

This dataset contains historical information about crawled data for different networks, including peer interactions, connection errors, protocols, multi-addresses, and timestamps. It provides a comprehensive view of these networks' structure and behavior, captured through regular crawls performed every 2 hours. The data is useful for researchers, developers, and network analysts interested in understanding peer connectivity, protocol usage, and network reliability. It includes both successful and failed connection attempts, along with detailed error information, making it a valuable resource for troubleshooting and optimizing network performance.

Each table in the dataset includes the crawling information for different networks.

Key Features

1. Crawl Identification

• crawl_id: A unique ID identifying the crawl that generated this datapoint. Crawls are performed every 2 hours.

2. Peer Information

• multi_hash: The multi-hash of the peer being crawled.
• agent_version: The agent version that this peer advertised. Can be null.
• peer_properties: Generic properties specific to certain networks.

3. Error Information

• connect_error: The error when trying to establish a connection to the peer.
• crawl_error: The error after successfully connecting to the peer and trying to drain its routing table.

4. Timestamps

• visit_started_at: The timestamp when the crawl started.
• visit_ended_at: The timestamp when the crawl finished.

5. Protocols and Addresses

• protocols: A sorted array of advertised protocols. Can be empty if the connection was not established or the libp2p identify exchange was not completed.
• multi_addresses: A sorted array of advertised multi-addresses used to connect to the peer.

Use Cases

1. Network Analysis: Analyze the structure and behavior of the Filecoin network, including peer distribution and protocol usage.
2. Error Analysis: Identify and troubleshoot common connection and crawl errors to improve network reliability.
3. Protocol Usage: Study the adoption and usage of different protocols within the Filecoin network.
4. Peer Behavior: Investigate patterns in peer behavior, including connectivity and activity over time.
5. Time-Series Analysis: Perform time-based analysis of network activity, error rates, and protocol adoption.
6. Tool Development: Develop tools and applications that leverage the data to improve Filecoin network monitoring and analysis.