Market Data for Traders: Guide

At a glance

Most strategies need: Clean daily or minute bars, point-in-time
Most overbought: Level 2 depth for strategies that do not use it
Most underinvested: Point-in-time universe and corporate action data
Storage default: Parquet files partitioned by symbol and year

Key takeaways

The data your strategy needs is determined by its holding period: slower strategies need clean history, faster ones need granularity.
Point-in-time universe membership and corporate actions matter more than data granularity for most systematic equity strategies.
Level 2 depth data is expensive and only useful for strategies that explicitly model order book dynamics.
Free data is adequate for liquid ETFs and index research and inadequate for anything requiring delisted instruments or point-in-time fundamentals.
Storing data in a columnar format and validating it on load saves more time than any optimisation of the research code.

Data types and what each is for

Type	Contents	Needed by	Relative cost
Daily bars	OHLCV per day	Swing, position, factor strategies	Low or free
Minute bars	OHLCV per minute	Intraday strategies	Low to moderate
Tick / trade data	Every transaction with timestamp	Microstructure research, execution analysis	High
Level 1 quotes	Best bid and ask	Execution modelling, spread measurement	Moderate
Level 2 depth	Full order book by price level	Market making, order flow strategies	High
Fundamentals	Financial statement data	Value, quality, factor strategies	Moderate to high
Corporate actions	Splits, dividends, mergers, delistings	Any equity strategy	Often bundled
Point-in-time universes	Index membership by date	Any cross-sectional strategy	Moderate
Reference data	Sector, listing, share count	Screening and neutralisation	Moderate

What each strategy type actually needs

Strategy	Minimum data	Also needed	Not needed
Trend following on futures	Daily bars, correct rolls	Contract specs, roll calendar	Tick data, depth
Equity momentum	Daily adjusted bars	Point-in-time universe, delistings	Intraday data
Mean reversion on ETFs	Daily bars	Dividend adjustments	Fundamentals, depth
Factor investing	Daily bars plus fundamentals	As-reported figures with timestamps	Intraday data
Intraday breakout	Minute bars	Session calendars, halts	Full depth
Market making	Full depth with timestamps	Message-level data	Fundamentals
Event driven	Daily bars plus event calendar	Announcement timestamps	Depth

Choosing a source

1
Determine whether you need delisted instruments
If your universe is individual stocks or crypto tokens, yes, and most free sources will not provide them. This single requirement usually determines the decision.
2
Check the adjustment methodology
What is adjusted, when, and whether unadjusted series are available. You will need both.
3
Verify timestamp semantics
Whether bars are labelled by open or close time, and in which timezone. Test against a known event.
4
Check the history length
Ten years is rarely enough for a slow strategy. Twenty to thirty is preferable, and some vendors only provide recent history cheaply.
5
Compare a sample against a second source
Differences reveal convention mismatches before they corrupt months of research.
6
Confirm the licence permits your use
Many retail feeds prohibit redistribution and some prohibit automated trading use. Check before building on them.

Storing and validating research data

STORAGE
   data/
     bars/daily/symbol=AAPL/year=2024/data.parquet
     bars/minute/symbol=AAPL/year=2024/month=03/data.parquet
     reference/universe_membership.parquet
     reference/corporate_actions.parquet

   Parquet with partitioning:
     - columnar, so loading two columns is fast
     - compressed, typically 5 to 10x smaller than CSV
     - preserves types, including timezone-aware timestamps

VALIDATION ON EVERY LOAD
   1. No duplicate timestamps
   2. Timestamps monotonically increasing
   3. high >= max(open, close) and low <= min(open, close)
   4. No negative prices or volumes
   5. No bar range more than 10x the trailing median
   6. No gaps in the expected trading calendar
   7. No prices repeated identically for many bars

Failing any check should raise, not warn.
Silent bad data is worse than no data.

A practical storage layout and validation routine.

Alternative data, briefly

Alternative data means non-traditional sources: satellite imagery, card transactions, web traffic, app downloads, and text. It receives substantial attention and is rarely the right investment for an individual.

It is expensive. Institutional datasets frequently cost more annually than a retail trading account holds.
Point-in-time integrity is difficult. Knowing exactly when a dataset became available is essential and often poorly documented.
History is short. Most alternative datasets have a few years of history, which is insufficient for validating a slow strategy.
Crowding is immediate. Commercially available datasets are bought by many funds simultaneously, which competes away the signal quickly.
Free versions exist. Public search trends, filings, and social data are accessible and correspondingly well mined.
The basics matter more. Clean point-in-time price and fundamental data, properly validated, offers more improvement to most retail research than any alternative dataset.

Frequently asked questions

Do I need tick data to backtest?

Only for strategies whose behaviour depends on intrabar sequence, such as scalping, market making, or precise stop-and-target modelling. For daily and swing strategies, clean daily bars with conservative intrabar assumptions are sufficient and far cheaper to obtain and store.

Is free market data good enough?

For liquid ETFs, indices, and major futures, generally yes. For individual stock universes requiring delisted instruments and point-in-time membership, no, because the bias introduced is systematically favourable. The decision usually comes down to whether your strategy selects among many instruments.

What is Level 2 data and do I need it?

Level 2 shows the full order book by price level rather than just the best bid and ask. It is necessary for market making and order flow strategies and unnecessary for anything holding positions for minutes or longer. It is one of the most commonly purchased and least commonly used datasets in retail trading.

How should I store historical market data?

Parquet files partitioned by symbol and time period for most retail-scale research: columnar, compressed, type-preserving, and fast to query. A database becomes worthwhile when you need concurrent access or complex joins, which most single-researcher workflows do not.

How much history do I need?

Enough to include several market regimes, which for daily strategies generally means at least fifteen to twenty years. Shorter histories cannot distinguish a strategy that works from one that suited a particular regime, and regimes commonly last five years or more.

Test this idea before you trade it

Describe the rules in plain language and AlgoTrader AI turns them into a structured strategy blueprint with a configurable historical backtest, cost assumptions, and exportable code.

Build a backtest

Keep reading

Referenced by

Educational use only. This guide explains how a strategy works. It is not investment advice, not a recommendation, and no result described here is a forecast. Test any approach on historical and out-of-sample data, size positions conservatively, and never risk money you cannot afford to lose.