An AI that repeats its answer can still be wrong
By OnTheRice · Sources and limitations · Report a correction
Source confidenceAuthor research judgment; not an outcome probability
What’s changing?
An AI judging study submitted on 8 October tested six models across four benchmarks. Its authors found that the most consistent judge matched the reference answers only 51% of the time. That figure describes the studied judge and tasks, not all AI answers.
Key points
- OTR's reading: repetition is a weak reason to trust a claim. When using AI for work, ask for the original source and check whether it supports the answer. A second assistant using the same source may repeat the same mistake.
What could change this view?
- These are author-reported research results. Different models and tasks may behave differently. The study does not measure the accuracy of every consumer chatbot or OTR's production system.
Bottom line
Check evidence as well as consistency. Watch whether AI tools make their sources easy to inspect and correct.
Its source facts and prices still need independent checks.
How likely is this call? · Three frameworks
This signal has no investible bullish or bearish call. A market-direction percentage does not apply.
Evidence behind this view
Checks whether separate pieces of evidence support the same view.
The arXiv record directly confirms the study's title, submission date, six-model/four-benchmark scope and limited meaning of the 51% result. The full study has not been independently replicated in this run.
What past price moves show
Compares today’s price moves with recorded past patterns.
No matched historical return study is supplied for this development.
Evidence against this view
Shows evidence and risks that could challenge this view.
These are author-reported research results. Different models and tasks may behave differently. The study does not measure the accuracy of every consumer chatbot or OTR's production system.
Why the direction changed
Explains recorded changes from expecting a rise to expecting a fall, or the reverse.
Learning review before publication
The supplied direction was retained: No investible directional call.
The supplied synthesis was retained.
Reason: Accepted after six actual collaborating reviews and deduplication against current Signals. Preserve attributed facts, conditional mechanisms, conflicting baseline disclosure and study-specific scope. No asset-return event is asserted; no probability or direction is assigned. Existing calls remain frozen.
Lessons considered: Keep the clocks separate; Compare matching quantities; Look for independent evidence; Keep incomplete tests visible; Separate direction from explanation; Use lessons in the next call.
This review belongs to the new draft before publication. Earlier published calls and their original deadlines remain unchanged.
What people say vs the facts
Compares recorded public discussion with checked market facts.
A steady answer can feel convincing. Accuracy requires a separate check against evidence.
How past forecasts performed
Shows recorded forecast results and what they can teach us.
No measured forecasting success rate is claimed for this article.
The pre-publication learning review is recorded under “Why the direction changed”, including cases where the supplied direction was retained.
Sources, original wording and price checks
Author-submitted research. The underlying facts and market prices have not been independently verified.
Original research record
An AI judging study submitted on 8 October tested six models across four benchmarks. Its authors found that the most consistent judge matched the reference answers only 51% of the time. That figure describes the studied judge and tasks, not all AI answers.
OTR's reading: repetition is a weak reason to trust a claim. When using AI for work, ask for the original source and check whether it supports the answer. A second assistant using the same source may repeat the same mistake.
These are author-reported research results. Different models and tasks may behave differently. The study does not measure the accuracy of every consumer chatbot or OTR's production system.
Check evidence as well as consistency. Watch whether AI tools make their sources easy to inspect and correct.
Forecast prepared Publication time unverified
No explicit investible BULLISH or BEARISH call on this card.
What changed
Recorded public historyAn AI judging study submitted on 8 October tested six models across four benchmarks. Its authors found that the most consistent judge…
This is the first recorded public view. Further public updates will appear here.
- An AI that repeats its answer can still be wrong
An AI judging study submitted on 8 October tested six models across four benchmarks. Its authors found that the most consistent judge matched the reference answers only 51% of the time. That figure describes the studied judge and tasks, not all AI answers.
Forecast prepared: 9 Oct, 17:16 SGT
Why the view changed
This is the first recorded view. Future source-backed updates will appear here.
Evidence, memory & review 1 linked source
Directional takeaway: N/A
OTR reasoning: An AI judging study submitted on 8 October tested six models across four benchmarks. Its authors found that the most consistent judge matched the reference answers only 51% of the time. That figure describes the studied judge and tasks, not all AI answers.
Counterpoint: These are author-reported research results. Different models and tasks may behave differently. The study does not measure the accuracy of every consumer chatbot or OTR's production system.
Source trail
- arxiv.org ↗Source date not independently verified
Review status: LEARNING REVIEWED WITH DISCLOSED LIMITS
No score or probability is presented.