Mid-training and post-training for language models
Models improve
while they work.
A scientific RL and SFT post-training library.
- Simulate thousands of situations, offline
- Grade every run, pass or fail
- Measure whether a change is real or noise
- Select the runs that teach, then train
Start here
Paste one prompt. Your coding agent builds the evals.
Works in Claude Code, Cursor and Codex. No GPU, no key. Training is step five, when you are ready.
Use whileai to build me better evals for my agent. 1. Install it: uv add whileai 2. Follow this skill: https://github.com/whilehq/whileai-sdk/blob/main/skills/strengthen-your-evals/SKILL.md 3. Read the docs at docs.withwhile.com when you need more. 4. You know my repo best: name the agent, behaviors and versions in its words. Rules: https://docs.withwhile.com/platform/naming 5. Before a second version, a sweep or a training run, follow https://docs.withwhile.com/platform/experiments: the question first, then per run five lines (Changed, Moved, Why, Learned, Reproduce), one chart, the rows that failed, points not fractions; then readback(tracked) reads the page the way I will.
The loop, as code
Simulate, grade, measure, select, train.
- 1Simulate. Throw thousands of situations at your agent, from broken tools to difficult users, offline and in a minute.
- 2Grade. Mark every run pass or fail with a test or a judge you have checked.
- 3Measure. Run both agents on the same tasks and find out whether the change is real, noise, or a regression.
- 4Select. Keep the prompts the model gets right only sometimes, because the ones it always or never solves teach nothing.
- 5Train. Run SFT or RL on what is left, with your trainer or ours.
uv add whileaiPython 3.10 to 3.13, two dependencies, typed. Docs.
import whileai as wai
@wai.tool
def get_order(order_id: str) -> dict:
"""Look up an order by id.""" # the function is the tool, its signature the schema
...
data = wai.simulate(
wai.seeded_agent([get_order]), # or your agent: any callable, or wai.OpenAI(...)
tools=[get_order],
system_prompt="Help customers with orders.",
simulator=False, # offline: no key, no network
mode="rl", # k rollouts per prompt, mixed groups filled first
situations=16,
repeats=4,
reproducible=True, # same draw every run
seed=0,
)A covering array over tools, world state and user stance. The world fails on purpose: a tool that times out, a customer who will not cooperate. Sixty seconds, offline.
Reproducing the literature
Try the newest RL and SFT ideas the week they publish.
Every paper here is one recipe: the paper's change against the same baseline, one GPU, under an hour, with an interval on the result. Two ideas held up. Three did not. That is what the library is for: pick up a new algorithm, run it on your own agent, and know whether it is real before you build on it.
Run one, or add oneGreen: yes. Gray: no, flat or a loss. Points over the baseline, whiskers are the 95% interval.
| Item | Value | 95% interval |
|---|---|---|
| Reward the answer, not the boxed format | +9.4 | +5.2 to +13.9 |
| Filter groups by outcome, not shaped score | +6.7 | +2.1 to +11.3 |
| Clip bound slides with the group | +5.0 | 0.0 to +10.0 |
| Endpoint SFT: keep the ends of the trace | -1.2 | -7.4 to +4.7 |
| Token selection by entropy x advantage | -31.0 | -38.3 to -23.5 |
Trained with the library
Four agents we trained with the library. Each card: out of 100 tasks, how many the agent got right before training and how many after.
Refund agent, GRPO
Before: 18After: 85Trained to look the order up before refunding, never invent an id.
Scored on 159 tasks it never saw in training. The gain is real: between +50 and +73 points.
Refund agent, DPO
Before: 17After: 92Same agent, trained on pairs of its own answers, two rounds.
Scored on 159 tasks it never saw in training. Round one gained +37 to +60 points, round two +17 to +35 more.
Text-to-SQL agent
Before: 53After: 74Trained to write SQL that runs on the customer's schema, every time.
Scored on 459 tasks it never saw in training. The gain is real: between +19 and +24 points.
Planted instruction
Before: 37After: 58Trained to ignore an instruction that arrives inside a tool result.
Scored on 119 tasks it never saw in training. The gain is real: between +15 and +29 points. Training on random examples gained nothing.
Mid-training and post-training
The improvement loop.
Simulate
simulate()seeded_agent()WorldOptions
Run your agent through many worlds, tools, users and faults, many times per prompt.
Grade
data.grade()verify.MathEqualverify.CodeExecverify.JSONSchema
Score every rollout, by a program where one can check the answer, by a judge where not.
Validate the judge
judge_trust()judge_probes()
Check the judge agrees with humans before you trust its scores.
Measure
pass_at()compare()eval_variance()holdout_size()
Compare two versions with intervals, so you know a gain is real.
Select
select()build_preference_pairs()curriculum()
Keep the rollouts that teach. Drop the ones every run gets right or wrong.
Guard
decontaminate()hack_scan()trace_markers()HackMonitor
Catch eval leakage and reward hacking before it reaches training.
Train and export
export()export_environment()platform.train()platform.serve()
Train on what you kept, in your stack or hosted, and serve the result.
Under the hood
Backed by science.
Every method and default cites the paper it came from, numbered in the references below.
Verifiable rewards [2]
When a program can check the answer, the reward is that program. A model judge only where no program can.
Intervals over tasks [3, 4, 5]
pass@1, pass^k and pass@k bootstrapped over tasks, not rollouts. holdout_size says how many prompts a gain needs.
Dynamic sampling [6, 7, 8]
Two rollouts per prompt first; only the prompts that disagree are filled to k. Selection keeps the 20 to 80% band.
The judge is an instrument [9, 10]
Agreement and Cohen's kappa against gold, length sensitivity, and never the model it is judging.
Over-optimization, expected [11, 12, 13]
The reward is a proxy and RL finds the gap. hack_scan looks for the feature that predicts reward against a shuffle floor.
Character from a spec [14, 15]
A constitution hashed into a version, a judge checked against the spec's own labels, length-matched pairs.
The defaults file every number, named, with its source or the words “convention, untested”
Every number the engine uses lives in one file, named, with the measurement or the paper it came from. Every one is a knob you can turn from the call. defaults.py
# DIFFICULTY_BAND = (0.2, 0.8): keep tasks the current policy passes between
# 20% and 80% of the time. rlhfbook.com/c/07-reasoning: difficulty filtering
# restricts RL prompts to those the starting model solves 20-80% of the time,
# measured from N=16 samples. DAPO (arXiv:2503.14476) is the online form.
DIFFICULTY_BAND = (0.2, 0.8)
# MIN_KAPPA = 0.6: chance-corrected agreement floor. Landis and Koch (1977)
# call 0.61-0.80 "substantial"; 0.6 is the bottom of that band. A 2026 sweep
# of 21 judges (arXiv:2606.19544) measured Cohen's kappa 0.376 to 0.511
# against human preference labels, so this floor is demanding on purpose.
MIN_KAPPA = 0.6
# DEFAULT_CONCURRENCY = 32: parallel rollouts, and the cap on parallel judge
# calls. One served vLLM replica's continuous-batching sweet spot for short
# chat turns; no throughput curve was recorded. (convention, untested)
DEFAULT_CONCURRENCY = 32The estimators pass@k, pass^k, the noise band
pass@k
Unbiased over n samples with c passing, per task, then averaged over tasks [3].
pass^k
All k repeats pass: the reliability line, not the headline.
noise band
The base is evaluated r times; a delta inside the re-run spread is not a gain [5].
References and how to cite 15 sources, one BibTeX entry
- 1Lambert, N. Reinforcement Learning from Human Feedback. arXiv:2504.12501, 2025.
- 2Lambert, N. et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124, 2024.
- 3Chen, M. et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, 2021.
- 4Efron, B., Tibshirani, R. J. An Introduction to the Bootstrap. Chapman & Hall, 1993.
- 5Miller, E. Adding Error Bars to Evals. arXiv:2411.00640, 2024.
- 6Shao, Z. et al. DeepSeekMath. arXiv:2402.03300, 2024.
- 7Yu, Q. et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025.
- 8He, J. et al. Skywork Open Reasoner 1 Technical Report. arXiv:2505.22312, 2025.
- 9Cohen, J. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20(1), 1960.
- 10Zheng, L. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS, 2023.
- 11Gao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML, 2023.
- 12Sharma, M. et al. Towards Understanding Sycophancy in Language Models. ICLR, 2024.
- 13Touvron, H. et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288, 2023.
- 14Bai, Y. et al. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073, 2022.
- 15Rafailov, R. et al. Direct Preference Optimization. NeurIPS, 2023.
- The full list, one per method, is in the README.
If the package is part of a result you publish. The repo carries a CITATION.cff as well.
@software{weiss2026whileai,
title = {whileai: post-training data and evaluation for tool-using agents},
author = {Weiss, Jacob},
year = {2026},
url = {https://github.com/whilehq/whileai-sdk}
}The platform, separate and optional
Hosted training and serving.
The library needs no account. Sign in once for hosted GPUs: the same rows push, train as LoRA SFT, GRPO, DPO or a reward model, and come back as a versioned OpenAI-compatible endpoint, with a paired verdict on the held-out set.
v1 = scored.push("refunds-v1", holdout=0.2, gate=True)
run = wai.platform.train(v1["datasetId"], method="grpo", steps=200)
run.wait()
model = wai.platform.serve("refunds-v2", run) # an OpenAI-compatible endpointBuilt in the open
Open source. Open models. Open datasets.
Get started
The next generation of agents
learns while they work.
Install the library, run the quickstart offline, then bring your model and your key. Sign in only when you want hosted training.