For verl users

Bring your verl experiments to a single consumer NVIDIA GPU. Keep familiar configs and Hydra overrides, connect your data and reward function, and run PPO, GRPO or OPD locally. This guide takes you from the first run to inspecting, resuming and exporting a model.

Start with the installed PPO/GRPO workflow

miniverl data sample --reward-profile target-length --rows 8 --out data/rl-prompts.parquet
miniverl run --example hydra-ppo --bind reward.provider=target_length --dry-run
miniverl run --example hydra-ppo --bind reward.provider=target_length --run-id local-ppo

This profile pins official verl v0.9.0 at 483b8a009ba3a97563edee3a19887e4862b8094a. Install miniverl[train,hydra] after CUDA PyTorch. Use --verl-config-path /path/to/verl/trainer/config --verl-config-name ppo_trainer followed by your usual dotted overrides to bring an existing experiment. Without a path, the pinned tree comes from the wheel. Hydra composition selects v5; resolved YAML still selects v4. Both retain the native IR as an inspection artifact, with PPO/GAE, GRPO, Dr.GRPO, RLOO and REINFORCE++ available. --print-resolved shows exactly what the compiler receives.

Follow the direct-config workflow to run, inspect, resume and export both examples. The real upstream corpus records complete configuration outcomes; the RL reference explains the algorithms and local execution model.

Command mapping

verl action	miniVERL action
compose config name + overrides	miniverl run --verl-config-name ppo_trainer algorithm.adv_estimator=grpo
use an already resolved config	miniverl run resolved-verl.yaml
inspect field semantics	miniverl run resolved-verl.yaml --dry-run --json
inspect the resolved local plan	read verl-direct-report.json and config.resolved.yaml
run actor/reward/reference phases	miniverl run resolved-verl.yaml --bind ...
read prompt Parquet	retain data.train_files and data.prompt_key directly
lower resource pools	record one process/device plus original source intent
inspect training and versions	miniverl inspect runs/<id> --json
recover an interrupted experiment	miniverl run resolved-verl.yaml --resume runs/<id> with the original bindings
export actor/critic and data	miniverl export-verl --run runs/<id> --target-verl v0.9.0 --out handoff

What maps into the RL runtime

• data.train_files, val_files, prompt_key, prompt/response limits, shuffle and seed drive the local Parquet source.
• data.train_batch_size × rollout.n is the logical trajectory count per rollout iteration.
• v4 retains v3's prompt-based actor.ppo_mini_batch_size, multiplying by rollout.n. Physical microbatch one and temporal offload are automatic.
• Actor model, revision, LoRA, optimizer, sampling, clipping and schedule fields feed the native recipe with their source units recorded.
• trainer.total_training_steps caps rollout iterations; total_epochs derives the schedule from the retained, filtered dataset and drop-last batches.
• Reward code uses an explicit local file/function binding plus checksum approval. Built-in exercise rewards and the upstream classifier-RM path are also available.
• Reference KL uses an independently frozen initial base. This role supplies the KL baseline, not a distillation teacher.

Distributed resource counts can be larger than one in the source config. They are classified distributed_only, preserved in the report, and lowered to one local process/device. Unknown algorithm or objective fields remain rejected.

Existing v0.8 OPD profiles

The OPD compiler targets official verl v0.8.0 at 7aed6b230776f963fa09509c10d9c3a767d1102c. It supports direct forward-top-k GKD and sampled-k1 policy-gradient distillation, including measured profiles and grouped/rewarded conformance variants.

miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
  --config verl-opd.yaml \
  --set 'data.train_files=["data/train.parquet"]' \
  --set actor_rollout_ref.actor.optim.lr=1e-5 \
  --accept-local-reinterpretations --out plan.json
miniverl run --profile verl-opd-v0.8-single-gpu-v1 --plan plan.json --dry-run

plan.json binds the source YAML, ordered overrides, scanned Parquet bytes, profile identity and native config. Use immutable plans and the OPD quickstart for that workflow.

Artifact handoff

Completed RL runs first export a standard adapter, then a pinned handoff bundle:

miniverl export-adapter --run runs/my-rl --out runs/my-rl/model
miniverl export-verl --run runs/my-rl --target-verl v0.9.0 --out scaleout
miniverl bridge doctor scaleout --json

The report separates artifact completeness, upstream config parse, model/data load smoke, reward implementation, launchability, distributed execution and algorithm parity. Read the RL workflow for the remaining cluster steps, or the OPD scale-out contract for the separately supported v0.8 distillation path.

Practical diagnostics

• Use miniverl doctor to inspect the CUDA stack.
• Resolve Hydra interpolation in the trusted upstream environment before importing.
• Keep immutable model and tokenizer revisions in portable configs.
• Reduce context or physical trajectory batch size on OOM; logical group and objective semantics remain unchanged.
• Read compatibility for field status and limitations for the consolidated boundaries.