Time-series prediction experiments

I built a custom NinjaTrader data extractor, but at the time I could access only three months of history. I looked for a longer, high-resolution time series to explore whether the data contained patterns a model could use and test them across different periods.

The work started before model training, with decisions about which data to obtain and how much history I could afford. It continued into a Python research pipeline using MNQ price and volume data, Polars for preparation and LightGBM for prediction. Claude Code handled implementation and runs under my direction.

What detail was worth acquiring?

Earlier acquisition work used Databento. I asked to see actual cost estimates before building the download pipeline, then asked how much one-second history the available credit could cover. More history would give the evaluation more periods to work through.

The choice was between individual transaction records and summaries of each second. The recorded estimates put two years of MNQ transaction data well beyond the budget. The acquisition used roughly two and a half years of one-second summaries instead.

Each summary, often called a bar, contains the opening, highest, lowest and closing price plus total volume. It preserves much more timing detail than daily data, while losing the sequence of individual transactions within that second. It also doesn't provide the number of transactions or the actual buying and selling activity separately. Those limits determine which questions the dataset can answer.

I had noticed differences between the NQ and MNQ activity data in the earlier work. In the later model project, I corrected an assumption that NQ should be the evaluation dataset: checks of the MNQ model needed more MNQ data. NQ could be a separate comparison later.

Make the input consistent, then question the resolution

The preparation code checks required fields, converts timestamps to UTC, sorts observations and removes duplicate timestamps. A separate inspection script reports date ranges, price ranges, average volume and duplicate counts. The target builder leaves an example out when its required future observation is missing or falls across a session boundary.

These checks help make the input consistent. They don't establish complete coverage. The acquisition notes flagged three degraded days, and those warnings need explicit treatment in a fresh evaluation. The small one-day file used to check that the pipeline runs also overlaps the later evaluation period; the research runner excludes it from the dataset.

The next question was how much of the original timing detail to retain. Combining five seconds into one row reduces the data to process, but could remove a pattern. I questioned that change and asked for comparisons: first five and ten seconds, then progressively wider intervals, eventually extending the search up to an hour.

Aggregation keeps the first opening price, the highest and lowest prices, the last closing price and summed volume for each interval. The selected configuration used 20-minute summaries to predict the next 20-minute price return. Starting with detailed data let us test that choice instead of committing to one interval at acquisition.

The interval search reused the research period. It selected a configuration; it didn't independently establish that the result would continue.

Test whether a prediction would be useful

For a finance application, a prediction must also survive the cost of acting on it. The evaluator selected the strongest positive and negative estimates as simulated trades. Tightening that selection changed the research result:

The primary assumption charged $1.50 for entering and leaving a trade. At $2.25, the research gain fell to about $397. That extra $0.75 across 2,605 simulated trades consumed $1,953.75. These are approximate historical simulation outputs; the conversion from returns to dollars uses a representative price.

Evaluation moved through time: train on six earlier months, evaluate on roughly the next month, then move forward and repeat. Within the September 2023 through November 2025 research dataset, 14 of 21 evaluation windows were positive under primary costs.

The selected configuration was then carried into three additional windows covering December 2025 through February 2026. All three lost money: about $737, $750 and $241. Together that was -$1,728 across 353 simulated trades, or about -$1,993 with the higher cost assumption.

The research notes had called the earlier result a breakthrough. The later entry says DO NOT TRADE this configuration as-is. The later data changed the conclusion.

What needs a cleaner evaluation?

The time split removes the prediction horizon plus an additional hour from the end of training. For the 20-minute prediction, that leaves an 80-minute gap before evaluation. This prevents training targets near the boundary from extending into that evaluation window.

Two implementation details still weaken that separation. The final model uses the later evaluation answers to decide when training stops. The evaluator also calculates its selection cutoffs from all eligible predictions in each evaluation window:

# Shortened illustration of the current selection rule.
upper_threshold = np.percentile(predictions, 97)
lower_threshold = np.percentile(predictions, 3)

At the start of that period, the rest of those predictions would not yet exist. A new evaluation needs training decisions and selection cutoffs determined using earlier information.

The existing 126 causality and time-split tests check useful properties, including whether adding future rows changes an earlier feature value. They don't resolve these dependencies or establish profitable execution.

For an AI team working with time-series data, the useful decisions here begin before the model: what information to buy, what aggregation removes, how to handle incomplete observations and what a later period can test. This experiment produced a rejected configuration and a clearer specification for the next evaluation.