AI document search should check what each cited number means
By OnTheRice · Sources and limitations · Report a correction
We connect a document-navigation study with a financial citation test. Better search can find the right area, while an unchanged number can still cite the wrong year. We would measure correct, source-supported answers alongside cost and valid answers withheld.
Source confidenceOriginal methods and reported results checked; independent replication absent
Published through Sites . The forecast window starts at its frozen preparation time above.
What’s changing?
We connect two studies surfaced in OTR Database Data. This is a new synthesis of earlier research, not a paper announced in the last 24 hours. RIT-RAG lets an AI search across document trees and choose which sections to read. On 150 FinanceBench questions using Claude Haiku 4.5, its authors report 84% answer accuracy, compared with 74% for Agentic Search. Mean online API costs are 10.1 versus 4.8 US cents per question, excluding one-time preprocessing. These are author-reported, AI-judged results. [Retrieval study](https://arxiv.org/html/2610.11370v1#S5)
Our new inference concerns the buying decision: a better search score is useful, but teams also need to know whether each cited number has the meaning the question requires. We connect the navigation result with a separate financial-evidence experiment that changes citations while leaving the calculation unchanged.
The citation study gives a clear example. A question asks about a 2010 index value. The 2010 and 2011 cells both contain 191, so moving the citation to the wrong year leaves the arithmetic unchanged. A number-only check cannot distinguish those roles. Other changed citations can express the same valid fact, so rejecting every changed location would also be wrong. [Matched citation examples](https://arxiv.org/html/2610.08675v1#S3.SS2)
We would therefore log three separate results for each answer: whether the calculation is right, whether the cited passage supports the requested company, period and quantity, and whether the system releases or withholds the answer. A passed calculation must not fill in the other two results.
The useful chain runs from document selection to section context, then to the cited cell and finally to the released answer. RIT-RAG addresses the first part by keeping surrounding headings and allowing another search. The citation study holds the supplied source and arithmetic fixed to probe the later link. Neither study evaluates the other system. [Search method](https://arxiv.org/html/2610.11370v1#S4) [Citation method](https://arxiv.org/html/2610.08675v1#S3.SS2)
Our proposed test joins these stages on the same held-out questions. Preserve the source revision, document path, table headers and actual cited location. Compare a simple number-at-location check with the added AI verifier. Measure what the verifier adds, including valid answers it rejects, before deciding that the extra step earns its cost.
Key points
- Keep calculation correctness, citation support and the decision to release an answer separate.
- Test identical numbers from different periods alongside different citations that genuinely support the same fact.
- Compare any added verifier with a simple number-at-location baseline, including cost and useful answers lost.
What could change this view?
RIT-RAG already reports passage precision and recall, as well as answer accuracy. Its results therefore include an evidence-retrieval check. However, reading a relevant passage and supporting every final cited operand are different tests. The paper also depends on usable document structure. [Retrieval analysis and limits](https://arxiv.org/html/2610.11370v1#S5.SS3)
The citation study's fresh constructed set has only 36 source pages and one non-author reviewer. In its 15 wrong-role pairs, explicit headers reduce accepted wrong alternatives from two to zero, but withheld valid originals rise from five to six. These are selected test cases, not real-world error rates. Earlier labels rely on assistant review, and the inspection packet was not publicly available at the paper's stated status. [Follow-up and limitations](https://arxiv.org/html/2610.08675v1#S3.SS4)
We have not run either system or combined them. Different models and tasks prevent a pooled success rate. More checks could waste time or block useful answers; a simpler system may already fit the intended job.
Bottom line
We would compare AI document-search products by the correct, source-supported answers they deliver at a stated cost and coverage. Keep the cited meaning beside the number, and keep harmless alternative evidence in the test set. Better navigation and stricter verification are promising tools, but their value depends on the full answer that reaches the user. No investment direction or measured customer saving is claimed.
Its source facts and prices still need independent checks.
How likely is this call? · Three frameworks
This signal has no investible bullish or bearish call. A market-direction percentage does not apply.
Evidence behind this view
Checks whether separate pieces of evidence support the same view.
OTR Database Data led us to two original studies. We keep each study as one origin cluster; its abstract, tables and appendix are not extra independent confirmations. The connection between navigation and final citation support is our proposed evaluation design, not a joint experiment already reported by the authors.
What past price moves show
Compares today’s price moves with recorded past patterns.
The earlier OTR article about an AI repeating a wrong answer concerns consistency across repeated judging. This view asks a different question: can an answer keep the same correct number while its evidence changes meaning? It adds a document-navigation method and a controlled citation test, rather than recounting the earlier judge result.
The financial study was submitted on 6 October. Its citation tests are later exploratory extensions of a prospectively collected question bank. We treat that sequence as research history, not a newly observed customer incident or an independent market forecast. [Study design](https://arxiv.org/html/2610.08675v1#S2)
Evidence against this view
Shows evidence and risks that could challenge this view.
RIT-RAG already reports passage precision and recall, as well as answer accuracy. Its results therefore include an evidence-retrieval check. However, reading a relevant passage and supporting every final cited operand are different tests. The paper also depends on usable document structure. [Retrieval analysis and limits](https://arxiv.org/html/2610.11370v1#S5.SS3)
The citation study's fresh constructed set has only 36 source pages and one non-author reviewer. In its 15 wrong-role pairs, explicit headers reduce accepted wrong alternatives from two to zero, but withheld valid originals rise from five to six. These are selected test cases, not real-world error rates. Earlier labels rely on assistant review, and the inspection packet was not publicly available at the paper's stated status. [Follow-up and limitations](https://arxiv.org/html/2610.08675v1#S3.SS4)
We have not run either system or combined them. Different models and tasks prevent a pooled success rate. More checks could waste time or block useful answers; a simpler system may already fit the intended job.
Why the direction changed
Explains recorded changes from expecting a rise to expecting a fall, or the reverse.
This new noninvestment research view preserves the earlier article about repeated AI judgments. It adds a distinct citation-meaning test. No frozen price call, probability, horizon or outcome changes.
What people say vs the facts
Compares recorded public discussion with checked market facts.
We have no sampled user discussion or adoption figures for these methods. Our view concerns an evaluation choice for teams using document-search tools; it is not a claim about public sentiment, widespread harm or customer uptake.
How past forecasts performed
Shows recorded forecast results and what they can teach us.
No market forecast applies. Study answer scores, verifier support scores and the number of answers a system releases cannot calibrate a future asset return. OTR has not measured the combined workflow proposed here.
Sources, original wording and price checks
Author-submitted research. The underlying facts and market prices have not been independently verified.
Forecast prepared
First published through Sites 09 Oct 2026, 19:44:43 SGT
No explicit investible BULLISH or BEARISH call on this card.
What changed
Recorded public historyWe connect a document-navigation study with a financial citation test. Better search can find the right area, while an unchanged number…
This is the first recorded public view. Further public updates will appear here.
- AI document search should check what each cited number means
We connect a document-navigation study with a financial citation test. Better search can find the right area, while an unchanged number can still cite the wrong year. We would measure correct, source-supported answers alongside cost and valid answers withheld.
Forecast prepared: 9 Oct, 19:22 SGTPublished by OTR: 9 Oct 2026, 19:44 SGT
Why the view changed
This is the first recorded view. Future source-backed updates will appear here.
Evidence, memory & review 2 linked sources
Directional takeaway: Not applicable: no investment call
OTR reasoning: We connect two studies surfaced in OTR Database Data. This is a new synthesis of earlier research, not a paper announced in the last 24 hours. RIT-RAG lets an AI search across document trees and choose which sections to read. On 150 FinanceBench questions using Claude Haiku 4.5, its authors report 84% answer accuracy, compared with 74% for Agentic Search. Mean online API costs are 10.1 versus 4.8 US cents per question, excluding one-time preprocessing. These are author-reported, AI-judged results. [Retrieval study](https://arxiv.org/html/2610.11370v1#S5) Our new inference concerns the buying decision: a better search score is useful, but teams also need to know whether each cited number has the meaning the question requires. We connect the navigation result with a separate financial-evidence experiment that changes citations while leaving the calculation unchanged. The citation study gives a clear example. A question asks about a 2010 index value. The 2010 and 2011 cells both contain 191, so moving the citation to the wrong year leaves the arithmetic unchanged. A number-only check cannot distinguish those roles. Other changed citations can express the same valid fact, so rejecting every changed location would also be wrong. [Matched citation examples](https://arxiv.org/html/2610.08675v1#S3.SS2) We would therefore log three separate results for each answer: whether the calculation is right, whether the cited passage supports the requested company, period and quantity, and whether the system releases or withholds the answer. A passed calculation must not fill in the other two results. The useful chain runs from document selection to section context, then to the cited cell and finally to the released answer. RIT-RAG addresses the first part by keeping surrounding headings and allowing another search. The citation study holds the supplied source and arithmetic fixed to probe the later link. Neither study evaluates the other system. [Search method](https://arxiv.org/html/2610.11370v1#S4) [Citation method](https://arxiv.org/html/2610.08675v1#S3.SS2) Our proposed test joins these stages on the same held-out questions. Preserve the source revision, document path, table headers and actual cited location. Compare a simple number-at-location check with the added AI verifier. Measure what the verifier adds, including valid answers it rejects, before deciding that the extra step earns its cost.
Counterpoint: RIT-RAG already reports passage precision and recall, as well as answer accuracy. Its results therefore include an evidence-retrieval check. However, reading a relevant passage and supporting every final cited operand are different tests. The paper also depends on usable document structure. [Retrieval analysis and limits](https://arxiv.org/html/2610.11370v1#S5.SS3) The citation study's fresh constructed set has only 36 source pages and one non-author reviewer. In its 15 wrong-role pairs, explicit headers reduce accepted wrong alternatives from two to zero, but withheld valid originals rise from five to six. These are selected test cases, not real-world error rates. Earlier labels rely on assistant review, and the inspection packet was not publicly available at the paper's stated status. [Follow-up and limitations](https://arxiv.org/html/2610.08675v1#S3.SS4) We have not run either system or combined them. Different models and tasks prevent a pooled success rate. More checks could waste time or block useful answers; a simpler system may already fit the intended job.
Source-grounded facts (paraphrased)
- 08 Oct 2026, 15:03:17 SGT: On 150 FinanceBench questions using Claude Haiku 4.5, RIT-RAG reports 84% versus 74% answer accuracy and 10.1 versus 4.8 US cents mean online API cost per question, excluding preprocessing, compared with Agentic Search. Source record: RIT-RAG-V1
- 07 Oct 2026, 00:57:47 SGT: The citation study holds arithmetic and operands fixed while changing a same-number source pointer; some changes alter financial role and some preserve equivalent facts. Source record: SAME-NUMBER-CITATIONS-V1
Source trail
- RIT-RAG: Navigating Document Corpora with Retrieval-Induced Trees ↗arXiv / study authors2026-10-08T07:03:17Z · published (UTC)
- Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge ↗arXiv / study authors2026-10-06T16:57:47Z · published (UTC)
Review status: PASSED
No score or probability is presented.