Qwen2.5 7b Tau2 Airline Sft LoRA

Qwen2.5-7B-Instruct · τ2-bench airline · distilled SFT LoRA

The RL warm-start checkpoint for multi-turn tool-agent RL on τ2-bench's airline domain.

Behavior-cloned from DeepSeek V4 Flash teacher trajectories, collected by running the teacher as the agent through the exact same τ2-bench harness used for training and evaluation.

The collection script requests deepseek-chat, which is DeepSeek's alias for their current chat model. Every raw API response in the training data reports model: deepseek-v4-flash (1156/1156 assistant turns, 671/671 user-simulator turns), so the teacher is V4 Flash, not V3. Stated here from the data rather than from the script.

Why this exists

GRPO on τ2-bench-airline from raw Qwen2.5-7B-Instruct is null — and not because of a bug. Base success is ~20%, so most rollout groups come back all-fail → zero intra-group variance → the group-relative advantage is ~0 → no gradient. RL sharpens what a policy already does sometimes; it cannot invent a skill. Worse, the base policy's own successful rollouts never invoke the write-tools (update_reservation_*) that held-out tasks require, so self-sampling can't bootstrap them either.

This adapter is the floor that makes the RL step work.

Results

This adapter, merged into the base, is the step 0 policy of the tau2_airline verl recipe. Measured under that recipe's protocol (held-out BINARY mean@4, 20 held-out airline tasks, VAL_TEMP=0.5, llama-3.3-70b user simulator):

	val@0 (this adapter)	val@20 (after GRPO)
seed 42	0.275	0.5625
seed 123	0.375	0.55

GRPO endpoint across 2 seeds: 0.556 ± 0.01. Measured evaluation noise on this same checkpoint (6 repeats of val@0) is 0.30 ± 0.05, so the two val@0 values above are a single checkpoint sampled twice from that band, not a difference.

Scope note. A separate hand-written pipeline reported base 0.20 → distilled SFT 0.405 → GRPO 0.545 for this approach, but under a different eval protocol (pass^1 over 10 trials with a self-hosted 7B user simulator, n=200). Those numbers are not directly comparable to the table above — different user simulator, different metric — and I have not re-verified that this exact file is the checkpoint they were measured on. The table above is what this file scores.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "yuyu0529nya/qwen2.5-7b-tau2-airline-sft-lora")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

To use it as an RL starting point, merge first (model.merge_and_unload()) and point the trainer at the merged weights — that is exactly how the recipe above consumes it.

This adapter only makes sense inside the τ2-bench airline harness — same system prompt, same 14 tool schemas, same dialogue loop. It is a policy for that environment, not a general chat model.

Training

base	Qwen/Qwen2.5-7B-Instruct
method	behavior-cloning SFT (QLoRA 4-bit) on teacher trajectories
teacher	DeepSeek V4 Flash (deepseek-v4-flash, requested via the deepseek-chat alias) — agent and user-simulator, temp 0.7
data	yuyu0529nya/tau2-airline-deepseek-distill (ds_114 config) — 114 successful teacher trajectories (reward = 1), deduped by action signature, capped 4/task
LoRA	r=16, alpha=32, dropout=0.05, targets = q,k,v,o,gate,up,down_proj
loss	assistant-token-only (render-twice-diff mask)

Teacher success on these tasks was ~76% vs base ~20%, and 51% of teacher trajectories invoke the write-tools the base policy never reaches on its own. That skill gap is why self-RFT from base cannot work here, and why distillation — rather than more RL — was the fix.

Licensing / provenance

• Base model: Qwen2.5-7B-Instruct, Apache-2.0 → derivatives unrestricted.
• Teacher outputs (DeepSeek V4 Flash): DeepSeek Open Platform ToS §4.2(2) assigns "any rights, ownership and interest" in outputs to the user; §4.2(3) explicitly permits applying inputs/outputs to "training other models (e.g. model distillation)".
• Task data: τ2-bench airline tasks are public benchmark data. All user profiles in the trajectories are synthetic (@example.com, RFC 2606 reserved). No real PII.
• Released under Apache-2.0.

Honest limits

• 20 held-out tasks. One task is worth 5 percentage points. Read single-point moves as noise.
• 30 training tasks, 114 teacher trajectories. Small-data regime throughout.
• Teacher/eval simulator mismatch: teacher dialogues used an API user-simulator; the reported evals use llama-3.3-70b. Not controlled for.
• airline domain only. Not evaluated on other τ2-bench domains. Not a general-purpose model.
• The reported runs used a hosted user simulator with provider fallback enabled, so they are not bit-reproducible.

Links

• Training data: https://huggingface.co/datasets/yuyu0529nya/tau2-airline-deepseek-distill
• verl recipe (PR): https://github.com/verl-project/verl-recipe/pull/121
• Project repo: https://github.com/yuyu0529nya/Yu-Agentic-RL
• τ2-bench: https://github.com/sierra-research/tau2-bench