On-policy distillation on one GPU

Generate with a student, score with a teacher and update the student, all on one NVIDIA GPU. Start from a built-in model pair or use your own verl profile.

The five profiles target verl v0.8.0 (7aed6b23). Their compatibility records distinguish the two measured systems workloads from grouped and rewarded extensions checked by conformance tests.

Start from the pinned profile

Install the matching CUDA PyTorch build for the machine first, then install the training and bitsandbytes dependencies:

python -m pip install "miniverl[train,cuda]"
miniverl data sample --format verl-parquet --out prompts.parquet
miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
  --config builtin:qwen3-0.6b-1.7b-opd \
  --set 'data.train_files=["prompts.parquet"]' --out plan.json
miniverl run --profile verl-opd-v0.8-single-gpu-v1 --plan plan.json --dry-run

Remove --dry-run only after inspecting the plan and confirming the model pair fits. External profiles must explicitly accept the reported high-risk local reinterpretations; the built-in profile has a value-bound approval manifest.

Runtime contract

Source intent	Local effect
actor dtype, quantization, attention	exact models.student runtime settings
teacher inference dtype plus miniVERL quantization/attention	exact models.teacher settings
data.train_batch_size	logical trajectories in one strict current-policy update
rollout batch/token fields	physical padded rollout limits
update trajectory/token fields	physical actor forward limits; never extra optimizer steps
lora_adapter_path plus pinned metadata	validated trainable student initialization

forward_kl_topk consumes teacher top-k token IDs and log-probabilities and reports top-k mass/overlap diagnostics. miniVERL's separate native bucketed_topk_tail objective adds an explicit K+1 tail bucket.

The second profile, verl-opd-v0.8-single-gpu-pg-k1-v1, records the sampled token, rollout-time actor log-probability and teacher log-probability, then recomputes the current actor log-probability for pinned k1 + vanilla policy-loss semantics. Its export carries sampled-token signals in place of a top-k target requirement. See Which profile should I use? for the choice guide.

verl-opd-v0.8-single-gpu-pg-k1-rewarded-v1 keeps that sampled-k1 path and adds an explicit deterministic task-reward advantage. The provider, group transform and both coefficients are versioned and carried by cache, checkpoint and export identity. This is a local task-plus-distillation objective, not a GRPO or distributed verl result.

Placement strategy

• NF4/int8 roles use resident local phases.
• Swap is available only for movable unquantized LoRA roles.
• Shared backbone requires compatible same-base roles.
• Unknown-size quantized roles use requires_probe until a bounded measurement establishes feasibility.

Normal planning is weight-free. plan --probe is a bounded, cached CUDA calibration with zero optimizer updates. The finalized plan therefore contains either a statically valid placement or the measurement needed to validate it.

The generated compatibility matrix and field-effect evidence are compiler-bound and byte-compared in CI.

Next: quickstart, hardware planning, or scale-out.