Skip to content

Real upstream compatibility coverage

The corpus starts from six upstream-owned launch examples at verl v0.9.0, commit 483b8a009ba3a97563edee3a19887e4862b8094a: PPO, GRPO, RLOO, REINFORCE++, GRPO LoRA and GRPO with a trained reward model. The documented Dr.GRPO overrides add a seventh configuration case, bound to the upstream README hash as well as the base launch script.

The generator reads literal launch arguments without executing shell commands, then uses Hydra 1.3.2 to compose the pinned upstream configuration tree. It records both the complete resolved configuration and the explicit launch fields, with independent compiler reports. Exact paths, hashes, deterministic environment bindings and per-example reasons are in the machine-readable manifest.

Native Hydra v5 outcomes

The new audit composes all seven official launch inputs through miniVERL's packaged upstream tree. Their resolved mappings equal the trusted upstream composition. Nine derived cases exercise shuffle, filtering, refill, legacy no-op fields and explicit rejection paths: 16 cases; 14 composed; 12 semantically accepted; 2 semantic rejections; 2 composition rejections.

The two semantic rejections are a missing filter metric and an unsupported policy-loss mode. Sweeps and unresolved references fail at composition. These counts are static evidence, not 12 executed model configurations; exact-model hardware is unassessed. The installed-wheel PPO/GRPO workflow separately measures native composition, updates, resume and handoff on the RTX 4080.

Native audit is regenerated by python scripts/publish_hydra_compatibility.py --check.

Historical direct-config v4 outcomes (preserved)

The v4 direct compiler accepts the experiment semantics of all seven complete official configurations. Every resolved leaf is classified. Local data and reward implementations are supplied as explicit runtime bindings; exact-model hardware capacity is assessed separately during execution.

Five derived stress configurations extend this audit: sequence-sum PPO and sampled validation pass; group filtering, oversampling and shuffled actor minibatches are explicit semantic rejections. Overall: 12 cases, 9 accepted, 3 rejected. These are static compilation outcomes. None of the 12 exact production-model configurations has been executed as part of this release, and their hardware status is unassessed—not zero blocked models.

The separate bounded PPO/GRPO qualification uses packaged upstream-format inputs and explicit length-reward bindings. It verifies the direct command, update counts, validation, exact resume and handoff on the installed candidate. See the direct workflow and the v4 machine-readable audit.

python scripts/publish_verl_direct_schema.py --check
python scripts/publish_direct_examples.py --check
python scripts/publish_direct_compatibility.py --check

Historical v3 outcomes (preserved)

Under v3, the original complete configurations reject: they include fields outside the documented compiler subset, such as automatic prompt filtering, upstream reward execution and distributed engine implementation settings. Required local reward, parameterization, reference and schedule choices are listed even when another field prevents import. The report separates counts of executable examples, examples needing local input, rejected examples, unimplemented fields and resource fields with a known lowering.

Historical v3 totals: 6 source examples, 7 cases, 0 original configurations locally executable, 7 rejected, 15 unimplemented field occurrences and 84 safely lowered resource-field occurrences. Dr.GRPO's recommended sequence-normalized loss and the upstream rollout-based reward model are explicit unsupported semantics, not silently replaced by the local token-mean loss or classifier RM.

The packaged PPO/GRPO workflows are explicit local adaptations, not extra successful upstream cases. Their ledgers record every changed or omitted upstream launch field, including the educational dataset, smaller actor, length reward, fixed iteration budget and disabled evaluation.

v3 profile units

verl-rl-v0.9-single-gpu-v3 keeps v2's PPO, critic, reward, KL and entropy capabilities and adds:

  • Upstream prompt minibatches expand by rollout.n into logical trajectory minibatches, for actor and critic alike.
  • GAE consumes algorithm.gamma; it is reported as an exact discount mapping.
  • Recognized distributed offload and dynamic-batching hints are recorded as physical local lowerings. Explicit local placement remains authoritative.

The evidence for minibatch units is upstream RayPPOTrainer._update_actor and _update_critic. Published v1/v2 rules and identities remain unchanged for existing artifacts. Use native Hydra v5 or resolved-file v4 for new upstream configs; v3 remains available for existing native-recipe workflows and does not reinterpret v2 artifacts in place.

Reproduce the corpus

git clone --depth 1 --branch v0.9.0 https://github.com/verl-project/verl.git upstream-verl
python scripts/publish_verl_compatibility_corpus.py --upstream upstream-verl --check
python scripts/publish_rl_product_examples.py --check

CI compares generated bytes against the committed corpus and tests that every resolved leaf receives one classification. Unknown objective fields remain rejected. Numerical and gradient conformance run separately against the same pinned upstream source. The last upstream drift audit found v0.9.0 still the latest stable release; published profiles are never silently retargeted.