The data-quality checklist for trading research

Bad timestamps, duplicated bars, missing sessions, and inconsistent symbols can create an edge that exists only in the dataset.

Know what each row means

Before calculating an indicator, document whether each row is a bar, tick, quote, trade, or aggregated event. Define the timezone, session boundary, price field, and whether values represent bid, ask, midpoint, or last trade.

A calculation can be mathematically correct while being economically wrong if its input meaning is unclear.

Run structural checks

Inspect ordering, duplicates, gaps, impossible highs and lows, zero or negative prices, extreme spreads, and timestamp jumps. Compare sample counts with a known market calendar and review suspicious periods visually.

Require monotonically increasing timestamps.
Check high is not below low and prices are finite.
Count missing intervals separately from legitimate closures.
Store the data source, retrieval date, and cleaning steps.

Treat cleaning as a research decision

Deleting an outlier, filling a gap, or stitching two feeds changes the evidence. Keep raw data immutable, write transformations as code, and compare results before and after cleaning.

A clean dataset is not one with no unusual events. It is one where unusual events are understood and handled consistently.

Discuss this article

Ask a setup question, share a backtest, or compare notes with other algorithmic traders. Use a display name; your email remains private.

Create an account or sign in above to join the conversation.

Loading discussion…