Agent Harness Wiki

4 min read

On-Policy Self-Distillation

Definition

On-policy self-distillation is a post-training loop in which the model first acts as a student in the ordinary task context, then the same model acts as a teacher after receiving extra context such as environment feedback, a demonstration, a runtime error, a unit-test failure, a reviewer comment, or a user follow-up. The teacher distribution becomes a dense learning signal for the student, usually through a KL-style distillation objective.

The important architectural move is that transient context-engineering becomes a candidate for durable behavioral change. A hint that would normally vanish with the context window can, in principle, be compressed into weights or adapters. This makes it adjacent to memory-persistence, but it is not the same as a note, retrieval store, or skill library.

Algorithm shape

A minimal loop looks like this:

  1. The student policy sees prompt or state X and produces an action/output Y.
  2. The environment returns extra context C: feedback, demonstration, tests, traces, user correction, or a richer judge explanation.
  3. The same model is run in teacher mode on X + C.
  4. The student is updated toward the teacher distribution, often with reverse KL or a related distillation loss.
  5. The resulting checkpoint or adapter is evaluated before promotion.

This differs from ordinary GRPO/RLVR-style training because the signal need not be one scalar reward for a whole rollout. If the feedback identifies why a code attempt failed, the teacher can change probability mass around the relevant token or decision rather than merely punishing every token in the failed response.

Why this matters for harnesses

For agent harnesses, self-distillation changes the status of feedback. In evaluation-and-review-loops, a failed unit test or reviewer note is already evidence. Under on-policy self-distillation, it can also become training material. The verifier is no longer merely a gatekeeper; it may become the producer of dense, replayable, credit-assignable feedback.

That makes feedback-rich environments a design target. rl-gyms-and-executable-environments-for-ai-harnesses should not only expose reward and terminal state; the next useful substrate exposes compiler errors, counterexamples, failed-test traces, reviewer rationales, and user corrections in forms suitable for replay and distillation.

Relationship to memory

Self-distillation is a form of parameter-persistent memory: behavior changes survive context reset because they are written into weights, LoRA adapters, or another trainable policy surface. That is more powerful than external memory, but also less inspectable. A serious harness therefore needs checkpoint lineage, adapter scope, rollback, evaluation gates, and privacy boundaries before treating it as normal memory.

It also complements self-evolving-workflows. Workflow evolution changes procedures, prompts, skills, or graph topology. On-policy self-distillation changes the agent policy itself. Those surfaces can reinforce one another, but conflating them would be a category error of the sort machines make just before inventing a mess.

Evidence from the 2026 video source

The 0xSero-linked Deep Learning with Yacine interview frames SDPO and SDFT as companion self-distillation methods:

  • SDPO applies the loop to RL settings with rich feedback and is claimed to reach GRPO accuracy about 6× faster while producing reasoning traces up to 11× shorter.
  • SDFT applies the same underlying idea to continual learning from demonstrations, documents, and interaction data, with less forgetting than ordinary SFT in the discussed experiments.
  • The speakers describe raw user conversations, runtime errors, failed tests, and user preferences as viable feedback sources.
  • The interview mentions OpenClaw RL and Continual Code as early examples of interactive agents using weight updates during operation, but this remains a lead until independently verified from primary system sources.

Design requirements

A harness that wants to use on-policy self-distillation needs more than a trainer:

  • Feedback capture: environment outputs must preserve why a rollout failed, not only whether it failed.
  • Replayability: training examples should reference stable traces, evidence records, and source artifacts.
  • Credit scope: the system should distinguish rollout-level, token-level, trace-node-level, and artifact-level feedback.
  • Promotion gates: adapted checkpoints or adapters should pass independent evaluation before becoming active.
  • Rollback: learned behavior can regress; promotion requires lineage and a way back.
  • Isolation: per-user or per-workspace adapters may be safer than global weight changes.
  • Governance: user-interaction logs require consent, filtering, privacy controls, and poisoning resistance.

Failure modes

Self-distillation is not a universal substitute for RL. The source explicitly warns that weak in-context learners make weak self-teachers. It may fail on small models or modalities without strong in-context learning, such as the robotics/VLA examples mentioned in the interview. When a model already succeeds often and the goal is just distribution sharpening, a simpler GRPO/RLVR regime may be preferable.

The teacher also lacks independent authority: it is the same model with more context. For that reason, self-distillation should consume external feedback from verifiers, users, tests, or environments; it should not replace independent review.

Read this with memory-persistence, context-engineering, evaluation-and-review-loops, self-evolving-workflows, rl-gyms-and-executable-environments-for-ai-harnesses, agent-facing-verifier-environment-architecture, and openclaw.

Graph View

Backlinks