Putting translation scores on one scale does not fix wrong rankings
By OnTheRice · Sources and limitations · Report a correction
Research reference · source time Not applicable: no market-price call · not a current quote or execution price
Source confidenceModerate for the reported methods; limited for use outside the tested tasks. We checked both original research abstracts. We have not reproduced either study.
Published through Sites . The forecast window starts at its frozen preparation time above.
What’s changing?
We see a new limit in attempts to make translation scores fair across writing systems. A COMET study separates two problems: scores can sit on different scales, and the system can still rank translations poorly within a writing system. Rescaling helps the first problem but cannot fix the second. Earlier research on cross-language AI confidence gives useful context, but is not new evidence today.
Key points
- We distinguish scale fairness from the ability to rank better and worse outputs.
- We also distinguish a useful ranking from a probability that matches how often answers are correct.
- We would keep original-language examples and human checks when comparing multilingual services. Neither paper proves a problem in every system.
What could change this view?
- We would narrow this concern if a held-out test showed that a particular service preserves meaning, ranking quality and confidence reliability across its intended languages. Different tasks, models and writing systems may behave differently.
Bottom line
We should ask what a multilingual score actually measures before using it to compare services. A rescaled score still needs checks against real correctness.
Its source facts and prices still need independent checks.
How likely is this call? · Three frameworks
This signal has no investible bullish or bearish call. A market-direction percentage does not apply.
Evidence behind this view
Checks whether separate pieces of evidence support the same view.
We checked two distinct research teams and tasks. The COMET paper tested five Indic languages while holding meaning and human ratings fixed. The probe paper used 3,000 maths problems, ten languages and eight model configurations. They support different parts of our interpretation, rather than repeating one test.
What past price moves show
Compares today’s price moves with recorded past patterns.
Not applicable: this is not a price forecast. Neither study tests asset returns.
Evidence against this view
Shows evidence and risks that could challenge this view.
COMET score normalisation improved pooled agreement with people, and multilingual probe training improved reliability. The studies therefore also show useful remedies. Their results do not establish the size of any problem in OTR or in a particular public service.
Why the direction changed
Explains recorded changes from expecting a rise to expecting a fall, or the reverse.
No investment call. An existing OTR language-coverage card cited the probe study. The new connection is the separate COMET finding: score normalisation can improve comparability while leaving within-script ranking errors. We do not count the reused paper as new independent evidence.
What people say vs the facts
Compares recorded public discussion with checked market facts.
We have no measured public-opinion sample for this claim. A vendor’s headline score should be checked against its task, languages and testing method; popularity does not validate a score.
How past forecasts performed
Shows recorded forecast results and what they can teach us.
We have no matched resolved forecast sample for this thesis. The papers report evaluation results, not an OTR prediction hit rate.
Sources, original wording and price checks
Author-submitted research. The underlying facts and market prices have not been independently verified.
Original research record
We focus on a distinction that a single headline score can hide. The new COMET paper reports both unequal score scales across writing systems and weaker ranking within each system. It shows that putting scores on one shared scale can improve comparison without repairing the ranking itself. We connect this with the already-covered finding that useful answer rankings do not guarantee reliable confidence.
Key points We distinguish scale fairness from the ability to rank better and worse outputs. We also distinguish a useful ranking from a probability that matches how often answers are correct. We would keep original-language examples and human checks when comparing multilingual services. Neither paper proves a problem in every system.
What could change this view? We would narrow this concern if a held-out test showed that a particular service preserves meaning, ranking quality and confidence reliability across its intended languages. Different tasks, models and writing systems may behave differently.
Bottom line We should ask what a multilingual score actually measures before using it to compare services. A rescaled score still needs checks against real correctness.
How likely is this call? · Three frameworks Market-direction probability: Not applicable. We make no price call. Bayesian reasoning would require new, task-matched evidence before transferring either study’s result to another system. De Finetti’s approach asks whether examples are comparable across tasks and languages; that assumption must be tested. Kolmogorov’s rules require a clear definition of success before assigning probabilities. A shared display scale, a correct ranking and a calibrated confidence score are different properties.
Evidence behind this view We checked two distinct research teams and tasks. The COMET paper tested five Indic languages while holding meaning and human ratings fixed. The probe paper used 3,000 maths problems, ten languages and eight model configurations. They support different parts of our interpretation, rather than repeating one test.
What past price moves show Not applicable: this is not a price forecast. Neither study tests asset returns.
Evidence against this view COMET score normalisation improved pooled agreement with people, and multilingual probe training improved reliability. The studies therefore also show useful remedies. Their results do not establish the size of any problem in OTR or in a particular public service.
Why the direction changed No investment call. An existing OTR language-coverage card cited the probe study. The new connection is the separate COMET finding: score normalisation can improve comparability while leaving within-script ranking errors. We do not count the reused paper as new independent evidence.
What people say vs the facts We have no measured public-opinion sample for this claim. A vendor’s headline score should be checked against its task, languages and testing method; popularity does not validate a score.
How past forecasts performed We have no matched resolved forecast sample for this thesis. The papers report evaluation results, not an OTR prediction hit rate.
Forecast prepared
First published through Sites 08 Oct 2026, 18:48:24 SGT
No explicit investible BULLISH or BEARISH call on this card.
What changed
Recorded public historyWe see a new limit in attempts to make translation scores fair across writing systems. A COMET study separates two problems: scores can…
This is the first recorded public view. Further public updates will appear here.
- Putting translation scores on one scale does not fix wrong rankings
We see a new limit in attempts to make translation scores fair across writing systems. A COMET study separates two problems: scores can sit on different scales, and the system can still rank translations poorly within a writing system. Rescaling helps the first problem but cannot fix the second. Earlier research on cross-language AI confidence gives useful context, but is not new evidence today.
Forecast prepared: 8 Oct, 18:37 SGTPublished by OTR: 8 Oct 2026, 18:48 SGT
Why the view changed
This is the first recorded view. Future source-backed updates will appear here.
Evidence, memory & review 2 linked sources
Directional takeaway: N/A
OTR reasoning: We see a new limit in attempts to make translation scores fair across writing systems. A COMET study separates two problems: scores can sit on different scales, and the system can still rank translations poorly within a writing system. Rescaling helps the first problem but cannot fix the second. Earlier research on cross-language AI confidence gives useful context, but is not new evidence today.
Counterpoint: We would narrow this concern if a held-out test showed that a particular service preserves meaning, ranking quality and confidence reliability across its intended languages. Different tasks, models and writing systems may behave differently.
Source trail
- Making COMET Comparable Across Scripts: Diagnosis and Correction of Tokeniser-Induced Script Bias in Indic MT Evaluation ↗arXiv primary research record2026-10-06T11:11:43Z · published (UTC)
- Cross-lingual Calibration of Pre-Generation Success Probes for Multilingual LLM Routing ↗arXiv primary research record2026-10-05T12:26:05Z · published (UTC)
Review status: SOURCE RESEARCH WITH DISCLOSED LIMITS
No score or probability is presented.