U.S. Commercial Aviation Industry Metrics

Context

Have you taken a flight in the U.S. in the past 15 years? If so, then you are a part of monthly data that the U.S. Department of Transportation's TranStats service makes available on various metrics for 15 U.S. airlines and 30 major U.S airports. Their website unfortunately does not include a method for easily downloading and sharing files. Furthermore, the source is built in ASP.NET, so extracting the data is rather cumbersome. To allow easier community access to this rich source of information, I scraped the metrics for every airline / airport combination and stored them in separate CSV files.

Occasionally, an airline doesn't serve a certain airport, or it didn't serve it for the entire duration that the data collection period covers*. In those cases, the data either doesn't exist or is typically too sparse to be of much use. As such, I've only uploaded complete files for airports that an airline served for the entire uninterrupted duration of the collection period. For these files, there should be 174 time series points for one or more of the nine columns below. I recommend any of the files for American, Delta, or United Airlines for outstanding examples of complete and robust airline data.

* No data for Atlas Air exists, and Virgin America commenced service in 2007, so no folders for either airline are included.

Content

There are 13 airlines that have at least one complete dataset. Each airline's folder includes CSV file(s) for each airport that are complete as defined by the above criteria. I've double-checked the files, but if you find one that violates the criteria, please point it out. The file names have the format "AIRLINE-AIRPORT.csv", where both AIRLINE and AIRPORT are IATA codes. For a full listing of the airlines and airports that the codes correspond to, check out the airline_codes.csv or airport_codes.csv files that are included, or perform a lookup here. Note that the data in each airport file represents metrics for flights that originated at the airport.

Among the 13 airlines in data.zip, there are a total of 161 individual datasets. There are also two special folders included - airlines_all_airports.csv and airports_all_airlines.csv. The first contains datasets for each airline aggregated over all airports, while the second contains datasets for each airport aggregated over all airlines. To preview a sample dataset, check out all_airlines_all_airports.csv, which contains industry-wide data.

Each file includes the following metrics for each month from October 2002 to March 2017:

1. Date (YYYY-MM-DD): All dates are set to the first of the month. The day value is just a placeholder and has no significance.
2. ASM_Domestic: Available Seat-Miles in thousands (000s). Number of domestic flights * Number of seats on each flight
3. ASM_International*: Available Seat-Miles in thousands (000s). Number of international flights * Number of seats on each flight
4. Flights_Domestic
5. Flights_International*
6. Passengers_Domestic
7. Passengers_International*
8. RPM_Domestic: Revenue Passenger-Miles in thousands (000s). Number of domestic flights * Number of paying passengers
9. RPM_International*: Revenue Passenger-Miles in thousands (000s). Number of international flights * Number of paying passengers

* Frequently contains missing values

Acknowledgements

Thanks to the U.S. Department of Transportation for collecting this data every month and making it publicly available to us all.

Source: https://www.transtats.bts.gov/Data_Elements.aspx

Inspiration

The airline / airport datasets are perfect for practicing and/or testing time series forecasting with classic statistical models such as autoregressive integrated moving average (ARIMA), or modern deep learning techniques such as long short-term memory (LSTM) networks. The datasets typically show evidence of trends, seasonality, and noise, so modeling and accurate forecasting can be challenging, but still more tractable than time series problems possessing more stochastic elements, e.g. stocks, currencies, commodities, etc. The source releases new data each month, so feel free to check your models' performances against new data as it comes out. I will update the files here every 3 to 6 months depending on how things go.

A future plan is to build a SQLite database so a vast array of queries can be run against the data. The data in it its current time series format is not conducive for this, so coming up with a workable structure for the tables is the first step towards this goal. If you have any suggestions for how I can improve the data presentation, or anything that you would like me to add, please let me know. Looking forward to seeing the questions that we can answer together!