TTPO: A New Method That Lets AI Models Learn Without Labeled Data at Test Time
Researchers from Zhejiang University have found a way to match supervised training performance — without using a single ground-truth label.
Table of Contents
One of the biggest bottlenecks in AI development is labeled data. Training a model to solve complex problems usually requires thousands of examples with correct answers — and creating those examples is expensive, time-consuming, and sometimes impossible.
A new paper from Zhejiang University’s ZJU-REAL group challenges that assumption. Their method, called TTPO (Test-Time Policy Optimization), shows that AI models can improve their performance during inference — the moment they’re actually being used — without any labeled training data.
The Problem TTPO Solves
When you want to make an AI model better at reasoning, you typically use reinforcement learning (RL) or self-distillation techniques. Both require ground-truth labels to work. At test time — when the model is deployed and answering real questions — those labels don’t exist.
The natural workaround is majority-vote pseudo-labels. If a model generates 10 answers to the same question and 7 of them agree, you treat that majority answer as the “correct” label. The problem is that this approach is fragile. One incorrect vote can corrupt the entire training signal.
The Key Insight
The TTPO team discovered something important: the failure mode is asymmetric. When a model’s rollout disagrees with the pseudo-label, it’s almost always wrong — regardless of whether the pseudo-label itself is correct.
This insight led to a clever solution. Instead of treating all rollouts equally, TTPO uses an asymmetric objective that handles agreeing and disagreeing rollouts differently.
How TTPO Works
Asymmetric Objective
The method splits model outputs into two branches based on whether they agree with the pseudo-label:
- Agreeing rollouts are distilled using On-Policy Self-Distillation (OPSD). The model learns from its own confident outputs.
- Disagreeing rollouts are penalized using Grouped Reinforcement Learning. The model learns that outputs contradicting the majority are usually wrong.
Token-Level Selection
Both branches are further refined at the token level. The distillation branch down-weights positions where the model has already converged, focusing learning on areas that still need improvement. The RL branch penalizes only confident errors — not uncertain ones — to avoid overcorrecting.
As the model improves, majority-vote routing yields tighter self-supervision, creating a virtuous cycle of improvement.
The Results
The numbers are striking:
- On Qwen3-1.7B, TTPO raised test-time training performance from 38.0% to 45.2%
- Without chain-of-thought thinking, improvements ranged from +25.2% to +36.4%
- Results were consistent across five competition-level math benchmarks
Critically, these results match what you’d get from label-supervised training — except TTPO doesn’t use any labels at all.
Why This Matters
The implications go beyond math benchmarks. If models can improve themselves at test time without human-provided labels, it opens the door to self-improving AI systems that get better the more they’re used.
This is particularly relevant for deployment in domains where labeled data is scarce or expensive to produce:
- Medical diagnosis — labeled cases are rare and require expert annotation
- Legal analysis — correct answers depend on jurisdiction and context
- Scientific research — ground truth often doesn’t exist for novel problems
What’s Next
The code is available on GitHub, and the paper is on arXiv. The ZJU-REAL group plans to explore whether TTPO generalizes beyond math reasoning to other domains like code generation and natural language understanding.
If it does, we might be looking at a fundamental shift in how AI models are trained and deployed.
