Dataset: Intraday Market Data

About this Dataset

Intraday Market Data

Context

There is a lot of buzz about institutional algorithmic trading on the stock market. There is also great interest in machine learning (here obviously). As an absolute beginner in ML with an interest in stock trading, I went looking for a market dataset that I could use.

While there are datasets available with daily closing prices, I couldn't find any publicly available with more granular data, much less anything close to having a real time data stream.

I have a trading account with TD Ameritrade. Their main desktop trading platform is called ThinkorSwim (TOS) and it has awesome capabilities. One feature is a Real Time Data (RTD) interface where Excel can pull price data from the platform at about a 3 second update rate. Using Visual Studio, I set up an Excel 'tunnel' to pull in data and then save it to a SQL Server DB. This dataset is that capture.

A version of the core data acquisition method is located at https://github.com/brtnsmth/TOS-RTD-core .

While the TOS RTD interface is one way (you cannot send trade orders to the platform), I can imagine a system that could deliver trading advice based on this near real time data.

Content

The datasets are 3 second interval price data, by week, using the futures trading windows - basically 24-7, from 6 PM EST Sunday to 6 PM Friday.

The example group includes futures, stocks and ETFs. No attempt was made to limit specific captures to 'market open' time periods.

There were also a few TOS refusal exceptions where data points may have been lost.

Acknowledgements

ThinkorSwim is an awesome trading platform. Excellent work.

Inspiration

Some basic data pattern analysis packages leading to trade recommendations for individual traders would be great. A multi-timescale momentum prediction tool?

Sharing

I have made this data freely available. I ask that others share HOW they are using this data to the depth and detail they are comfortable with.

Tables

Tos Kaggle Data Example

@kaggle.brtnsmth_intraday_market_data.tos_kaggle_data_example

14.82 KB
20 rows
22 columns


CREATE TABLE tos_kaggle_data_example (
  "id" BIGINT,
  "timestamp" TIMESTAMP,
  "n__es" DOUBLE,
  "n__nq" DOUBLE,
  "n__rty" DOUBLE,
  "spy" DOUBLE,
  "qqq" DOUBLE,
  "iwm" DOUBLE,
  "aapl" DOUBLE,
  "msft" DOUBLE,
  "nvda" DOUBLE,
  "xlk" DOUBLE,
  "xlf" DOUBLE,
  "xlp" DOUBLE,
  "xly" DOUBLE,
  "xtn" DOUBLE,
  "hyg" DOUBLE,
  "db_col_16" BIGINT,
  "db_col_17" BIGINT,
  "n__es_volume" BIGINT,
  "tlt" DOUBLE,
  "tlt_volume" DOUBLE
);

Tos Kaggle Data Week Ending 2025–01–10

@kaggle.brtnsmth_intraday_market_data.tos_kaggle_data_week_ending_2025_01_10

3.65 MB
143707 rows
27 columns


CREATE TABLE tos_kaggle_data_week_ending_2025_01_10 (
  "id" BIGINT,
  "timestamp" TIMESTAMP,
  "n__es" DOUBLE,
  "n__nq" DOUBLE,
  "n__rty" DOUBLE,
  "spy" DOUBLE,
  "qqq" DOUBLE,
  "iwm" DOUBLE,
  "aapl" DOUBLE,
  "msft" DOUBLE,
  "nvda" DOUBLE,
  "xlk" DOUBLE,
  "xlf" DOUBLE,
  "xlp" DOUBLE,
  "xly" DOUBLE,
  "xtn" DOUBLE,
  "hyg" DOUBLE,
  "tnx" DOUBLE,
  "tyx" DOUBLE,
  "n__es_volume" BIGINT,
  "tlt" DOUBLE,
  "tlt_volume" DOUBLE,
  "vix" DOUBLE,
  "spx" DOUBLE,
  "spx_pcr" DOUBLE,
  "spy_pcr" DOUBLE,
  "n__es_pcr" DOUBLE
);