Missing data, outliers, and bad timestamps

A practical process for finding the data defects that silently change signals, session filters, and backtest outcomes.

Separate absence from zero

A missing bar is not the same as a bar with zero volume or unchanged price. Filling missing values can create artificial continuity and signals that never existed. Keep a missingness mask so later analysis knows which values were observed and which were inferred.

Investigate unusual values

Extreme prices, spreads, durations, and volumes may be genuine events or feed errors. Compare them with neighbouring records, another source, and the broker's session. Do not remove a difficult event merely because it harms the result.

Flag outliers instead of deleting them immediately.
Record the reason for every correction.
Run the strategy both with and without disputed records.
Review whether the EA would have been able to trade the event.

Timestamps are part of the data

A one-hour timezone error can move a session filter, daily reset, or news window onto the wrong candles. Store raw timestamps with their timezone and convert only at the display or rule boundary.

Test daylight-saving transitions explicitly. A strategy should not quietly change behaviour twice a year because a hard-coded offset was used.

Discuss this article

Ask a setup question, share a backtest, or compare notes with other algorithmic traders. Use a display name; your email remains private.

Create an account or sign in above to join the conversation.

Loading discussion…