Adding AI to a forecast needs proof of extra value

Original research record

We connect two research questions. The first asks whether an AI model adds useful information beyond a market, crowd or statistical forecast. The second asks how the reward used in training changes a forecaster’s errors. Together, they make model selection a test of extra value for a defined job. We would compare the original forecast with the combined forecast on the same unseen cases, using rules chosen before the results.

Key points We distinguish being accurate alone from improving an already useful forecast. We would keep the strong-market result visible: the competence gate did not significantly improve that subset. We would examine confidence reliability and costly mistakes alongside the main score. We treat these older studies as newly connected research, not a new discovery announced in the last 24 hours.

What could change this view? We cannot assume these results transfer to every forecasting team. The competence study excluded retrieval and used historical questions; contamination risk remains. Its gate also did not significantly beat the strong market subset or a strong contextual stacking baseline. The reward study used one base model and one dataset, with limited repeated training runs. Neither paper proves an improvement in OTR.

Bottom line We would add forecasting models when a fair test shows useful extra information. Agreement, a confident explanation or a larger team is not enough by itself.

How likely is this call? · Three frameworks Market-direction probability: Not applicable. We make no bullish or bearish asset call. Bayesian updating needs evidence tied to a defined question; no fitted likelihoods are supplied here. De Finetti’s approach asks whether the cases and model errors are genuinely comparable. Kolmogorov’s rules keep probabilities consistent, but consistency alone does not prove that adding a model improves a forecast.

Evidence behind this view We checked two primary research records and their relevant methods, results and limitations. Competence-Gated Pooling reports 2,357 binary questions and five models. Its main baseline Brier score fell from 0.0771 to 0.0732; lower is better. This is a result for that test pool, not a universal gain. The separate scoring-rule study tests five training rewards. [Competence study](https://arxiv.org/abs/2609.12101) [Reward study](https://arxiv.org/abs/2608.28482v2)

What past price moves show Not applicable: we make no asset-price forecast. Event-forecast benchmark results cannot supply a trading history or a price pattern for an unnamed asset.

Evidence against this view We cannot assume these results transfer to every forecasting team. The competence study excluded retrieval and used historical questions; contamination risk remains. Its gate also did not significantly beat the strong market subset or a strong contextual stacking baseline. The reward study used one base model and one dataset, with limited repeated training runs. Neither paper proves an improvement in OTR.

Why the direction changed No investment direction applies and no locked forecast is changed. We found neither study’s identifier nor this baseline-addition mechanism in the 343-record publication model. This is a research-method connection, not a claim that OTR has installed or validated it.

What people say vs the facts We have no public-opinion sample for this finding. The competence study’s self-reported confidence test concerned four Qwen models; it does not show that every form of model uncertainty fails. We should test performance rather than count confident-sounding answers.

How past forecasts performed We have no resolved OTR sample for this proposal. The reward paper finds different confidence and error profiles despite similar overall performance; its small aggregate differences are sensitive to training seed. Those research results are not an OTR hit rate or forecast probability.