Training web agents to use tools and search the internet effectively is one of the hardest open problems in AI. The fundamental dilemma is this: how do you get an agent to improve from its own experience when the only feedback you have is whether it found the right answer at the end of a long chain of actions? Supervised fine-tuning (SFT) requires expensive teacher-generated trajectories that are fixed and limited. Reinforcement learning (RL) with sparse rewards provides almost no signal for the intermediate decisions that matter most. A new paper from HKUST presents an elegant solution: DeepSearch-Evolve, a self-distillation framework where an agent iteratively generates its own training data, filters it, and learns from it — no teacher model required.
Contents
The Challenge of Training Web Agents
Modern web agents need to plan, search, read pages, reflect on what they found, verify their progress, and recover from dead ends. Training them traditionally relies on one of two approaches:
- Supervised Fine-Tuning (SFT) — You collect expert trajectories (from a teacher model or human demonstrations) and train the agent to imitate them. This works but is fundamentally capped by the quality and diversity of the fixed training data. The agent never learns to do better than its teacher.
- Reinforcement Learning (RL) — You let the agent explore and reward it for correct final answers. But in long-horizon tasks spanning dozens of tool-use steps, the reward signal is extremely sparse. The agent gets a +1 at the end and has no idea which of its 30 actions actually mattered.
The key insight from the HKUST team is that both approaches can be combined in a self-sustaining loop if you have the right environment — one that is deterministic, verifiable, and rich enough for the agent to practice on at scale.
What is DeepSearch-World?
DeepSearch-World is the environment that makes self-distillation possible. It is an offline Wikipedia-based simulation with three critical properties:
- Deterministic — Given the same actions, the environment produces the same observations every time, making trajectory comparison and filtering reliable.
- Verifiable — Answer correctness can be checked automatically, providing the objective signal needed for rejection sampling.
- Reproducible — Search results and page contents are indexed offline, so results are consistent across runs.
The environment contains 420,000 multi-hop QA tasks constructed from entity-level random walks through Wikipedia. Each task requires the agent to follow a chain of connected entities and use search and page-reading tools to find the answer. The tool set includes:
web_search_wiki— A Pyserini BM25 index over Wikipedia for text searchvisit_wiki— A SQLite offset database backed by JSONL corpus for full-page reading
These tools mirror real web search but run entirely offline, making them fast, deterministic, and free from the cost and variability of live API calls.
The Self-Distillation Framework: DeepSearch-Evolve
The core contribution of the paper is the DeepSearch-Evolve training loop, an iterative process that turns a base model into a continually improving agent:
Iterative self-distillation loop:
1. Scaffold teacher trajectory generation (plan → act → reflect → state update)
2. Rejection sampling (LLM Judge checks correctness)
3. Trajectory quality filtering (remove weak evidence, weak alignment)
4. Scaffold-to-ReAct conversion (format into ShareGPT)
5. Full SFT via LlamaFactory
6. Data mixing with decay-based importance sampling
7. Repeat → 15 rounds total1. Scaffold Teacher Trajectory Generation
Each trajectory is generated using a structured scaffold format that explicitly separates planning, action, reflection, and state tracking. This is more structured than the standard ReAct format and is designed to teach the model specific cognitive behaviors:
- Progress Verification — The agent tracks step-by-step progress toward the goal.
- Grounded Reflection — The agent reflects on observations and adjusts its approach.
- Failure Recovery — When a path leads nowhere, the agent recognizes the dead end and backtracks.
2. Rejection Sampling and Quality Filtering
Not all generated trajectories are useful. The framework applies a two-stage filter:
- Rejection Sampling — An LLM Judge checks whether the final answer is correct. Only correct trajectories pass.
- Quality Filtering — Trajectories with weak evidence chains or poor alignment between reasoning and tool-use are also discarded.
In the main experiment, 10,000 trajectories are generated per round, and training is triggered once 4,000 correct trajectories are accumulated.
3. Scaffold-to-ReAct Conversion and Training
After filtering, the scaffold-format trajectories are converted to the ReAct format (which is more natural for inference) and stored in ShareGPT format for training. Full SFT is performed using LlamaFactory with DeepSpeed ZeRO-3, Flash Attention 2, and bf16 precision at 32K context length.
4. Data Mixing with Decay-Based Importance Sampling
A key design choice is how data from different rounds is mixed. The team uses decay-based importance sampling — trajectories from earlier rounds are downsampled relative to newer ones. This prevents the model from overfitting to lower-quality early trajectories while still retaining diversity.
Benchmark Results
The final model, DeepSearch-World-9B (based on Qwen 2.5 9B, trained with Qwen3.5 template), was evaluated on three benchmarks:
| Benchmark | DeepSearch-World-9B | Context / Comparison |
|---|---|---|
| BrowseComp | 31.2% | Qwen3.5-27B: 61.0% | DeepSeek V4 Flash Max: 73.2% | Top: 90%+ |
| GAIA | 61.5% | OxyGent (OSS): ~59% | Enterprise ensembles: 90%+ |
| HotpotQA | 93.4% | Near SOTA for multi-hop QA in distractor setting |
The numbers tell a compelling story. At 61.5% on GAIA, DeepSearch-World-9B surpasses published open-source agents like OxyGent (~59%) despite being a 9B model without any teacher distillation. The 93.4% on HotpotQA is particularly impressive — it approaches state-of-the-art for multi-hop reasoning in the distractor setting. On BrowseComp (31.2%), while well below frontier models (the best systems reach 90%+ with multi-model ensembles), the result is competitive for a standalone 9B model and demonstrates that self-distillation produces meaningful improvements over the base.
Why This Matters
The significance of DeepSearch-World goes beyond the numbers. Three aspects stand out:
1. Self-Evolution Without Teachers
The model improves purely from its own generated trajectories, filtered by correctness. No GPT-4, no Claude, no human annotators generating gold-standard demonstrations. This is a scalable path to improvement that could run indefinitely — as long as there are more tasks in the environment, the agent can keep generating training data and learning from it. The paper demonstrates this over 15 evolutionary rounds, with sustained improvement across all benchmarks.
2. Fully Open Source
The project is fully open-sourced at github.com/ornamentt/DeepSearch-World:
- The DeepSearch-World environment with 420K multi-hop tasks
- Validation set and evaluation harness
- Model weights (DeepSearch-World-9B)
- Full training code using LlamaFactory as a submodule
3. The Verifiable Environment as a Scaling Strategy
The paper demonstrates that a relatively simple environment — offline Wikipedia with search and page-reading — is sufficient to bootstrap complex agentic behaviors through self-distillation. This suggests that as long as you can create a verifiable environment with sufficient task diversity, self-evolution is a viable training strategy. The architecture is general: the same approach could be applied to code execution environments, math environments, or any domain where correctness is checkable.
Limitations and Future Directions
The results are not without caveats. The BrowseComp score of 31.2% shows that self-distillation from a 9B base model still has a significant gap to frontier systems. The environment, while rich with 420K tasks, is limited to Wikipedia — real web search involves heterogeneous sources, dynamic content, and multimodal information. Extending the self-distillation framework to live web environments, more diverse domains, and larger base models are natural next steps. The HKUST team has laid the groundwork for a paradigm where agents improve themselves through practice, which may prove to be one of the most important directions in web agent research.
Conclusion
DeepSearch-World and DeepSearch-Evolve present a clean, well-executed answer to a fundamental problem: how to train tool-use agents to improve from their own experience without expensive supervision. By combining a verifiable offline environment with an iterative self-distillation loop, the HKUST team shows that a 9B model can reach competitive performance on GAIA, BrowseComp, and HotpotQA entirely through self-generated data. The approach is general, fully open-sourced, and points toward a future where agents grow smarter simply by practicing in environments where truth is verifiable.
Paper: DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment (Xinyu Geng, Xuanhua He, Sixiang Chen, Yanjing Xiao, Fan Zhang, Shijue Huang, Haitao Mi, Zhenwen Liang, Tianqing Fang, Yi R. Fung — HKUST)
Code and data: github.com/ornamentt/DeepSearch-World
