ADR 0010: pinned verl v0.8 sampled-k1 policy-gradient contract

Status: accepted for verl-opd-v0.8-single-gpu-pg-k1-v1.

Decision

miniVERL implements one closed, reward-free policy-gradient OPD profile pinned to verl v0.8.0 commit 7aed6b230776f963fa09509c10d9c3a767d1102c. The contract is the composition of upstream kl_penalty_forward(..., "k1") and compute_policy_loss_vanilla: rollout-time actor log probability minus teacher log probability forms the sampled k1 estimator, its detached negative is the token advantage, and the current/old actor probability ratio is clipped with the pinned vanilla settings before token-mean aggregation.

The cache binds sampled token IDs, old actor and teacher log probabilities, response mask, actor policy version, teacher/tokenizer identity, profile identity and estimator implementation version. A target cannot cross policy versions. Each accepted group is generated by the current policy and produces exactly one optimizer update.

Boundary

This is not PPO. It has no task reward, critic, externally supplied advantage, reference KL, replay, multiple generations, multi-teacher routing, multimodal input or distributed execution. Only loss_mode: k1, use_policy_gradient: true, policy_loss_mode: vanilla, n: 1, and the conformance-tested clipping values are accepted. Other upstream options fail closed instead of being approximated.

Verification

The optional conformance suite loads the exact pinned upstream function bodies and compares estimator tensors, detached advantages, scalar loss, metrics, actor-logit gradients and a deterministic optimizer step. Runtime evidence is systems/conformance evidence only; it makes no task-quality or alignment claim.