FALCA: Falsification-Aware Reinforcement Learning with Hierarchical Credit Assignment for Long-Horizon Deep Search Agents

Method

Searcher–Falsifier research loop

At each research stage the Searcher proposes candidate hypotheses from its current research state. For each candidate, an independent Falsifier follows a contradiction-first strategy: it searches for counterevidence and returns a structured report through the refute tool, with a verdict (reject, survive, or uncertain), refuted and unresolved constraints, retrieved evidence, and recommended follow-up queries. Rejected candidates are removed, unresolved constraints become new search targets, and evidence is preserved through update_context, which also marks the boundary of a research segment. Both agents share the same search and visit tools, so they reason over consistent evidence.

Falsification-guided SFT warmup

Trajectories generated under this protocol are filtered by rejection sampling, keeping those with correct answers, valid tool interactions, and sufficient evidence support. SFT on these trajectories, with environment observations masked, teaches the Searcher tool use, context updating, Searcher–Falsifier coordination, and candidate revision before reinforcement learning.

Hierarchical reward and credit assignment

Binary final-answer rewards cannot tell critical actions from redundant exploration. FALCA rewards a trajectory at three granularities and assigns each signal to the policy tokens it describes: a trajectory reward for final answer correctness, broadcast across the trajectory and normalized by the number of segments; a segment reward that spreads the credit for each completed sub-goal uniformly over the research stages that led to it; and a turn reward from an LLM judge that rates each action as harmful, neutral, useful, or critical, including whether it falsifies an unsupported hypothesis or triggers a meaningful research-state revision.

Policy optimization without group normalization

For hard queries, all rollouts of a prompt often receive the same final reward, and group-relative advantages vanish. FALCA therefore optimizes the token-level shaped advantages directly with a PPO-style clipped objective, using DAPO-style asymmetric clipping and a KL penalty towards the SFT reference policy. Rollouts run asynchronously, and turn-level process scores are computed as soon as each trajectory finishes, so slow pages or long falsification rounds do not block the batch.

Results

Bold marks the best result under the Falsifier-based harness.

The Falsifier helps, and training is what makes it pay off

With the Qwen3.5-4B backbone unchanged, switching to the Falsifier-based harness raises the average score from 51.8 to 55.5. Under the same harness, FALCA-4B raises it further to 64.7 and beats the untrained Qwen3.5-4B on every benchmark: SFT teaches the propose–falsify protocol, and agentic RL teaches the Searcher how to act on falsification reports. FALCA-4B also surpasses Qwen3.5-9B under the same harness (64.7 vs. 57.9) and remains competitive with several 30B–32B search agents.

The largest gains are on BrowseComp, xbench-2510, and DeepSearchQA, which demand multi-step evidence aggregation, constraint verification, and candidate elimination. On WideSearch-en, which emphasizes broad-coverage collection, the gain is smaller, while the absolute score stays strong at 65.8.

Analysis

Shaped advantages under degenerate rollout groups

About 66% of rollout groups in our training have zero within-group reward variance, which removes the learning signal of GRPO-style group-relative advantages. Shaped advantages keep learning from segment- and turn-level differences: the training reward keeps rising instead of plateauing early, and the agent searches more persistently.

Case Studies

Falsification reports act as state transitions: they change what the Searcher believes and where it searches next, rather than only checking the final answer.

BibTeX

@article{dou2026falca,
  title   = {FALCA: Falsification-Aware Reinforcement Learning with
             Hierarchical Credit Assignment for Long-Horizon Deep Search Agents},
  author  = {Dou, Zijun and Luo, Kun and Li, Chaofan and
             Lu, Shuqi and Liu, Zheng},
  journal = {arXiv preprint arXiv:2610.28361},
  year    = {2026}
}