RecoAtlas is a benchmark and RL environment for LLM recommendation agents. One episode works like this:
- The agent gets a shopping request and a number K. For example: "I'm putting together a polished casual summer outfit with a comfortable short-sleeved top, relaxed high-waisted shorts, and a statement waist accent to define the silhouette." K is how many products it must recommend.
- It searches a fixed catalog with three tools: 104,993 H&M clothing products or 62,961
Amazon Musical Instruments products.
search_products(query, top_k): thetop_kproducts closest to a text query;get_complementary_products(item_ids, top_k): products that complement the given products, from other subcategories (learned from co-purchases in the default "utility" tools);get_substitute_products(item_ids): splits a list into kept products and removed near-duplicates (same subcategory, very similar).
- It submits a report of K products, each with a short reason. The paper parser removes duplicate, unknown and unobserved IDs, then fills missing positions from earlier discoveries.
- The report is compared with the request's reference items: K products taken from one real customer's purchases. The request was written to describe those products. K is between 3 and 7.
sethit: the share of the reference items found among the first K submitted products, from 0 to 1.valid: 1 if the report has exactly K different catalog products, each returned earlier bysearch_productsorget_complementary_products. Otherwise 0.reward_direct:sethitif valid, otherwise −1. This is the RL reward.
Validity and rewards are computed after filtering/backfilling, in every RL adapter. Backfilling uses discovery order, never reference labels. Malformed schemas fail without backfilling. If too few candidates were discovered, the processed report can remain short; it receives partial SetHit but an invalid-report RL reward. A missing/failed report has zero SetHit and reward −1. SetHit and the training reward are separate signals.
From a clone of this repo:
uv sync --extra toolsimport recoatlas
env = recoatlas.load("hm-clothing", device="cuda") # or device="cpu"
# Or: recoatlas.load("amazon-musical-instruments", device="cuda")- First call: downloads about 2.8 GB into the Hugging Face cache and checks every file.
- Later calls: about 6 seconds.
- Memory: about 5 GB of RAM.
ep = env.episode(split="agent_hidden", index=0)
print(ep.request) # the shopping request
print(ep.k) # how many products to submit
ep.search_products("relaxed high-waisted shorts", top_k=10) # returns JSON text
ep.get_complementary_products(["0722674001"], top_k=5)
picks = ep.seen_ids[: ep.k] # products the tools have returned so far
print(ep.submit([{"product_id": p, "reasoning": "fits the outfit"} for p in picks]))
print(ep.score()) # {'sethit': ..., 'valid': 1, 'reward_direct': ...}submitreturns the processed report underreport, plus label-freevalidanderrors. Raw submissions are retained inep.submissions. Rewards and references are not returned bysubmit.- You can submit again. The last report is the one scored.
Every request in a split, with the built-in no-LLM baseline:
from concurrent.futures import ThreadPoolExecutor
from recoatlas.baselines import run_baseline
n = len(env.splits["agent_hidden"]) # 1,960 requests
with ThreadPoolExecutor(32) as pool:
scores = list(pool.map(lambda i: run_baseline(env.episode("agent_hidden", index=i)), range(n)))
print(sum(s["sethit"] for s in scores) / n)All threads share one env, so the tools are loaded once.
Install the inference client, tools and frozen grammar compiler:
uv sync --extra agent --extra grammarYou need a running compatible model endpoint. The paper configuration is packaged in
src/recoatlas/configs/paper_agent.json; it records the Qwen checkpoint/revision, temperature and
thinking limits. Local vLLM serving profiles cover Qwen, Gemma 4, Nemotron 3.5 Lightning and
GLM-4.7-Flash. The server must support compact structured_outputs.grammar, the selected model's
chat-template parameters, and (for thinking on) thinking_token_budget with the matching reasoning
parser. This package does not start or download the agent model. Loading Env downloads only
the recommendation-tool model and artifacts.
Configure the endpoint and, if it requires authentication, a key:
export OPENAI_BASE_URL=http://127.0.0.1:8001/v1
export OPENAI_API_KEY=EMPTY # only for an unauthenticated local endpointUse your actual key for an authenticated endpoint; do not commit it. .env files are not loaded
automatically. Start with one development query:
uv run recoatlas-agent --dataset hm-clothing --split agent_dev --limit 1 \
--device cuda --thinking off --output runs/smoke.jsonlThen evaluate the hidden split (omit --limit to include every query):
uv run recoatlas-agent --dataset hm-clothing --split agent_hidden \
--device cuda --thinking off --tools utility --output runs/utility-off.jsonl
uv run recoatlas-agent --dataset hm-clothing --split agent_hidden \
--device cuda --thinking off --tools semantic --output runs/semantic-off.jsonlUse --device cpu without a GPU; this controls the recommendation tools, not the remote agent.
Use --model to select the served model and --thinking on for reasoning. Both modes send an
explicit thinking switch; they do not rely on the server's default. The output limit is 1,000
tokens with thinking off and 16,384 with thinking on, including a 4,096-token reasoning budget
in the latter. Temperature is 1 in both modes. These limits are shared across the local profiles.
The following names select their profiles automatically (case-insensitive). For another served
alias, supply --model-profile explicitly. A recognized model with the wrong profile is rejected.
Unrecognized local models are rejected before downloading tools or making requests; selecting a
profile for an alias asserts that its underlying model supports that interface.
| Model identifier | Profile | vLLM --reasoning-parser |
|---|---|---|
Qwen/Qwen3.8-27B-FP8 |
qwen |
qwen3 |
google/gemma-4-31B-it |
gemma4 |
gemma4 |
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 |
nemotron |
nemotron_v3 |
zai-org/GLM-4.7-Flash |
glm4_7 |
glm45 |
The Qwen profile also recognizes the paper's Qwen3.5 size variants and Qwen3-4B/8B. An explicit profile is not a compatibility guarantee for other model generations or quantizations.
Inspect the resolved controls without an API key, downloads, files or inference:
uv run recoatlas-agent --model google/gemma-4-31B-it --thinking on --dry-runThe output includes required_server.reasoning_parser and required_server.reasoning_config.
Use those in your vLLM server's --reasoning-parser and --reasoning-config options. For example,
on hardware large enough for Gemma 4 (adjust parallelism and context capacity to your hardware):
vllm serve google/gemma-4-31B-it --port 8001 --max-model-len 32768 \
--reasoning-parser gemma4 \
--reasoning-config '{"reasoning_start_str":"<|channel>","reasoning_end_str":"<channel|>"}'Install a vLLM build that supports that checkpoint, parser, thinking budget and structured-output
combination; the serving installation is separate from this package's client dependencies. The
model's chat template must accept enable_thinking. See the linked official sources below for
model-specific installation requirements. This agent emits JSON actions, not native function
calls: do not replace its prompt or enable a different tool-calling agent loop.
Run one development query against the server, then repeat with --thinking off and a new output:
uv run recoatlas-agent --model google/gemma-4-31B-it --thinking on \
--split agent_dev --limit 1 --output runs/gemma-on-smoke.jsonlIf the server uses a custom alias, e.g. --served-model-name my-local-model, use:
uv run recoatlas-agent --model my-local-model --model-profile gemma4 --thinking off \
--split agent_dev --limit 1 --output runs/gemma-off-smoke.jsonlInspect the saved model calls: reasoning should be separate from final JSON. Responses with unparsed reasoning delimiters, missing final content, exhausted thinking output limits, or nonempty reasoning while thinking is off are rejected. Absence of reasoning in a particular thinking-on response is not itself an error. Client tests do not prove that a server honors the budget: validate both modes on the actual serving installation before running a full split.
Provenance: Qwen request controls retain the original release behavior. The other profiles
implement documented interfaces from vLLM's Gemma guide,
the Nemotron model card,
and the GLM model card and
chat template.
Budgeting uses vLLM's thinking-budget interface.
The packaged configs/open_models.json identifies all four checkpoint revisions, 65,536-token
contexts and the recorded vLLM build. Not every historical server
argument was recorded. The profiles have not been live-inference validated in this package.
They provide compatibility, not evidence of exact reproduction of the published figure. Keep
checkpoint/tokenizer revisions, vLLM version, launch arguments and server sampling defaults with
your run; a standalone recoatlas-agent client does not discover or verify those settings.
scripts/run_open_models.sh runs Qwen3.8-27B-FP8, Gemma 4 31B IT, Nemotron 3.5 Lightning
30B-A3B BF16 and GLM-4.7-Flash BF16. Each runs with thinking off and on, on MI and H&M:
16 arms, 1,960 hidden queries each, 31,360 evaluated episodes in total. This is fresh inference,
not replaying old predictions. No RL training, GPT baseline, LLM judge, HF upload or Git push runs.
Where to run it: on a Linux GPU host already allocated to you, with H200/B200 GPUs visible to the process. This script does not allocate Ray clusters or attach GPUs to your login machine. GPU capacity in a queue is not an allocation. A Ray job must run this command on a GPU worker, not its CPU-only driver. Use a persistent mounted directory for results on preemptable workers; node-local output will not survive loss of that node. The script does not copy to HDFS.
-
Install the client in the checkout:
uv sync --frozen --extra agent --extra grammar
-
Install the serving environment separately, on the GPU host. Python 3.12 and an NVIDIA driver compatible with the installed CUDA/PyTorch build are needed. The exact recorded vLLM wheel is available from its full-commit index:
uv venv --python 3.12 .venv-serving uv pip install --python .venv-serving/bin/python \ --extra-index-url https://wheels.vllm.ai/8263ea12bd8fa7584277f5200d521193259707b7 \ 'vllm==0.29.1rc1.dev134+g8263ea12b' 'transformers==5.17.0' 'xgrammar==0.2.6' ninja
The launcher refuses a different vLLM version and records the serving package versions. It starts authenticated loopback-only servers using temporary local keys. No paid API key is needed. All pinned model and tool repositories are public; HF write tokens are not passed to child processes. Downloads still need network access.
-
Inspect the plan; this does not start models or download artifacts:
bash scripts/run_open_models.sh --dry-run --gpu-groups 0 1 2 3,4 \ --server-python "$PWD/.venv-serving/bin/python"Here GPUs 0, 1 and 2 serve Qwen, Gemma and Nemotron respectively; GPUs 3 and 4 serve GLM with tensor parallelism 2. Groups must be disjoint and on the same host. Use the physical IDs/UUIDs in your allocation's
CUDA_VISIBLE_DEVICES, not renumbered indices. With only two allocated GPUs, use--gpu-groups 0,1: models then run sequentially. The script never uses GPUs outside the groups you supply. Allocate enough local scratch for all four checkpoints, tool files and compilation caches; allow roughly 300 GB plus room for the traces. -
On that GPU host, run the full sweep in
tmux,screen, or your batch scheduler:bash scripts/run_open_models.sh \ --server-python "$PWD/.venv-serving/bin/python" \ --gpu-groups 0 1 2 3,4 --concurrency 8 \ --output /path/to/persistent/open-models-mi-hm \ --scratch /path/to/large-local-disk/recoatlas-cache \ --max-hours 14Replace both example paths. Each model first runs the first development query of each size K=3–7 on both datasets in both modes (20 smoke episodes per model). A failed smoke, invalid report, thinking-off reasoning, or no observed thinking-on reasoning blocks hidden evaluation for that model. Other models continue.
--smoke-onlystops after these development checks. Smoke gates never use recovery scores to select configurations. They check transport and behavior; a sample cannot prove budget enforcement for every request. Missed prompted tool minima after budget exhaustion are recorded, not smoke failures: this preserves the original paper agent's behavior rather than filtering its policy outcomes.The frozen agent, six-round budget, three tools, parser, output limits and temperature remain unchanged. Recommendation tools use CUDA in float32 with search batching disabled. The concurrency default is 8 per server; the historical manifests recorded 32. Context is 65,536, thinking budget 4,096, and output limits 1,000 (off) / 16,384 (on). Single-GPU serving except GLM, 0.85 memory fraction and server seed 0 are explicit launcher settings, not claims that every historical launch argument was recovered. Different hardware/sampling may give different scores.
-
Inspect
OUTPUT/shard-0-of-1/summary.json. Each model directory containsserver.log,runtime.json,status.json, andsmoke/andhidden/subdirectories. Under each dataset:thinking-on.jsonl/thinking-off.jsonl: reports and traces in dataset order;*.summary.json: completion counts, pending queries, provenance and mean SetHit;*.jsonl.state/: compressed, atomic per-query checkpoints; keep these for resume;*.log: progress and errors from that arm.
Per-query checkpoints are saved immediately. JSONL exports are assembled at arm completion; after a hard interruption, resume reconstructs them. Scores include failed queries as zero; an incomplete arm's mean is
null, not a partial result labeled as a full benchmark. The sweep exits nonzero for failures or interruption. On its time limit or Ctrl-C it stops only the child processes it started; it does not destroy your cluster or release its reservation. -
To continue, repeat the same command with
--resume. Code, configuration, selected rows, revisions and runtime must match. Already recorded successes and errors are preserved; only missing IDs run. A failed development check is not retried until it passes by chance: investigate it and use a separate new output for a deliberate new run. You can first use--smoke-only, inspect it, then use--resumewithout--smoke-onlyfor the full evaluation.
For multiple allocated hosts, run the same script on each with a shared --num-shards N and a
different --shard-index from 0 to N−1. Rows are assigned by their fixed split position modulo N,
identically in both modes. Each host still runs all four models and needs a two-GPU group for GLM.
Keep every shard-I-of-N output directory. Shard summaries are not full-benchmark scores: combine
all N disjoint shards and weight by their query counts, not an unweighted mean of shard means.
This is explicit sharding, not automatic Ray provisioning or a claim to use all free cluster GPUs.
Use --models qwen (or gemma4, nemotron, glm4_7) for a model-specific replica. Then only
that model's GPU group is required; use the same shard count across all replicas of that model.
A hosted endpoint can use --hosted --json-only --model YOUR_MODEL. This omits local controls:
reasoning remains provider-managed (thinking: null in the run summary), even with the default
--thinking off. --hosted --thinking on and hosted model-profile overrides are rejected.
The frozen strategy and task contract stay fixed.
Each run writes reports, tool traces, model calls, protocol counts and post-report SetHit to JSONL,
plus a .summary.json containing the resolved model profile, client controls and required server
settings. The Qwen revision is not attributed to other models. Failed queries stay in the denominator
with zero SetHit. Existing output requires explicit --resume; configuration mismatches are
rejected. --concurrency runs independent agents over shared, unbatched tools. --limit results
are subset results, not full-benchmark results.
The Python entry point is Agent(env, backend).run(request, k); there is only one agent
implementation. Six decision rounds, two distinct searches and one complementarity call are
agent-side requirements. The near-duplicate call at 0.95 is requested but not forced. Its result
is advisory: it does not delete candidates or override the final selection. The environment itself
does not impose the agent's budget on arbitrary RL policies.
The direct-relevance judge gives one 0–4 score per recommended set, normalized by four. It sees the request, K and anonymized catalog descriptions (300 characters each), not reference items, purchase history, agent explanations, tool condition, retrieval scores or SetHit.
First validate saved predictions without calling an API:
uv run --extra llm recoatlas-judge --traces runs/utility-off.jsonl --dry-runThen set OPENAI_BASE_URL and OPENAI_API_KEY for your judge endpoint (which can differ from the
agent endpoint), and explicitly select an accessible deployment:
uv run --extra llm recoatlas-judge --traces runs/utility-off.jsonl \
--model YOUR_JUDGE_DEPLOYMENT --output runs/utility-off-judge.jsonlThe paper deployment alias is gpt-5.6-luna; using another model is recorded as such. For traces
from agent_dev, add --split agent_dev. Judging incurs API charges and is never launched
automatically by an agent rollout or reward computation. Up to three attempts are recorded per
report; the first valid response is used. Failed judgments remain unresolved, not zero. Failed
agent reports are skipped; an incomplete report marked complete is rejected during preflight.
Judging requires only the llm extra and catalog/query files, not embeddings or a GPU.
For each request, your agent:
- reads
ep.requestandep.k; - calls tools with
ep.call(tool_name, **arguments). The tool descriptions, for a prompt or a tool schema, are inrecoatlas.env.TOOL_DOCS; - finishes with
ep.submit(results).
Then average ep.score()["sethit"] over the split.
-
TRL: pass
recoatlas.trl.RecoAtlasEnvironmenttoGRPOTrainerasenvironment_factory. Seeexamples/trl_grpo.py. -
OpenEnv-based trainers: start the server, one process for all episodes:
uv sync --extra server RECOATLAS_DEVICE=cuda uv run recoatlas-openenv --port 8000
-
Miles:
recoatlas.miles.MilesToolServiceis the tool service for a Miles RecoAtlas agent loop. Seeexamples/miles_agent.py.
Pass them to recoatlas.load(...). The adapters read them from RECOATLAS_<NAME> environment
variables instead, e.g. RECOATLAS_DEVICE=cuda.
| setting | default | what it does |
|---|---|---|
tools |
"utility" |
"utility": search and complementary tools fine-tuned on purchase data. "semantic": the pretrained embeddings without that fine-tuning. |
max_tool_results |
None |
optional cap for custom experiments; the paper agent requires no environment cap |
tool_noise |
0.0 |
share of complementary and near-duplicate results deliberately corrupted (by default, swapped for a random product of the same subcategory), to test robustness to faulty tools |
device |
"cpu" |
"cpu" or "cuda" |
batch_search |
False |
True: searches from parallel episodes share one encoder pass, about 4× faster; scores then vary by up to 0.0001 |
The deterministic baseline asks for max(20, 5*K) candidates. The paper agent forwards each
chosen top_k (the compact grammar permits 1–100); it is not silently reduced to 20 or to the
baseline's count. Its prompt-visible ledger holds 60 candidates. Paper-agent runs reject capped
or batched environments. The existing tool-noise setting remains available for controlled
corruption experiments; it does not change the frozen agent prompt.
- Internals, design choices, and measured speed and memory: docs/overview.md.
- Install options and adapters: README.md.