This blog is authored by Assaf Gerner, Guy Rephaeli, Asaf Karnieli, Guy Freund, Shaked Meirom and Joanna Kramer, in collaboration with Alice’s Liran Cohen and Dadi Atar.

Over just three weeks, our Forward-Deployed Researchers (FDRs) worked alongside Alice’s research team to post-train the recently released NVIDIA Nemotron 3.5 Super VL to improve its resilience to attacks such as indirect prompt injection and malicious requests. The results demonstrate how specialized open models can surpass frontier-level quality at a fraction of the cost: 

  • 41-point improvement on attack resilience as compared to the base model
  • 10-point improvement as compared to GPT-6.1 Sol
  • 25x cheaper than GPT-6.1 Sol (cost per 1M tokens)
Figure 1: Our training recipe lifted Nemotron 3.5 Super VL from comparable in quality and cost to GPT-6.1 Luna to surpassing GPT-6.1 Sol in quality at a fraction of the cost. 

This post details Alice’s use case and presents the custom training recipe our FDRs developed to achieve the frontier-level results above. 

Improving agent resilience to attacks with Alice 

Alice, an AI safety and security company, builds RL Environments for post training that are informed by real-world intelligence on threat actor behavior (the “Rabbit Hole”) and input from 150+ staff security researchers and expert network members. They provided AI21 with a set of RL gyms to train agents to defend against security threats across their enterprise environments, including platforms, systems, tools, data, and workflows, such as Slack and GitHub. The goal is to produce agents that score highly on both resistance to attacks and task completion  – so that one aspect does not come at the expense of the other. The goal is to produce agents that score highly on both resistance to attacks and task completion – so one aspect does not come at the expense of the other.

Figure 2: An example Slack task from Alice’s gym. Company and domain names are fictional.

One kind of these attacks is indirect prompt injection – malicious instructions planted in information an agent reads with tools. For example, an agent might be prompted to send a password reset reminder to employees on Slack – but the reset link is corrupted, risking exposing employee login credentials. An agent trained in Alice’s RL gyms will learn to detect that breach and prevent it, while still accomplishing its original task: notifying employees to reset their passwords (Figure 2).

As more enterprises integrate agents into their core workspaces – which entails high levels of access to sensitive information, as well as widespread access to employees – ensuring that these agents are equipped to spot and stop attacks becomes paramount. Our work post-training Nemotron 3.5 Super VL on Alice’s RL gyms proves that it’s possible for an agent to maintain high attack resilience alongside task completion.  

The challenge

To successfully post-train Nemotron 3.5 Super to excel on this use case, the FDR team had to address two core challenges. 

The first challenge was how to improve attack resilience without hurting task completion on regular benign tasks. To take the example from above, if the Slack agent refuses to stop working after it detects the malicious reset link, it has successfully defended the company – yet is ultimately not a very useful agent. 

The second challenge was around data. Given that the training dataset included several hundred examples, the team needed to find a way to train without overfitting the model to the training data.


Our FDR team carefully designed the post-training process to resolve these challenges. 

Methodology

Training the agent inside Alice’s RL gyms

Addressing both challenges meant training Nemotron 3.5 Super the way it will actually be used: as an agent acting inside a live environment, not as a model answering isolated prompts. Alice’s gyms are packaged as Harbor tasks [1], and each task comes with its own container image holding a simulated workspace, such as Slack or GitHub. For every rollout, the AI21 Training Platform launches a fresh, isolated sandbox from that image. The model then works through the task inside the sandbox: it reads messages, calls tools and takes actions. When the episode ends, the gym scores the outcome. Giving each rollout its own sandbox keeps training secure, because anything the agent does in response to an attack stays inside the sandbox. We use this same setup throughout: to generate the trajectories for SFT, to produce the live rollouts for GRPO, and to evaluate every checkpoint.

Warm-starting with near-policy SFT

Online reinforcement learning has become the standard approach for post-training language models. Because it is on-policy, the model learns from rollouts it generates during training rather than from a dataset collected in advance. Training on its own outputs improves quality and reduces the risk of catastrophic forgetting. The most widely used online RL method, GRPO, samples a group of rollouts for each prompt and compares them, so the model learns what separates better outputs from worse ones. This means GRPO learns only when the rollouts in a group receive different rewards. If every rollout in a group gets the same score, every advantage is zero, the group contributes no gradient, and nothing is learned.

Figure 3: How surprised the base model is by each model’s rollouts. We score each set of rollouts by its perplexity under the Nemotron base model. Lower perplexity means the rollouts are closer to the base model’s own distribution. Values are shown relative to the base model’s perplexity on its own rollouts (reference). Average OS Model: the mean over rollouts from open-source models whose rollouts served as SFT data. Original SFT: rollouts from the policy fine-tuned on those rollouts as-is. Rephrased SFT: rollouts from the policy fine-tuned on the rephrased versions. Error bars show 95% CIs.

However, when the FDR team ran the initial evaluation of Nemotron 3.5 Super, they found that the rewards of its rollouts rarely varied within a group. Before they could move to RL, they first ran a supervised fine-tuning (SFT) stage to raise the fraction of tasks with non-zero reward variance within a group [2]. After SFT, the share of training tasks with non-zero reward standard deviation rose from 35% to 66%.

The FDR team generated successful rollouts on the gyms from several open models, yet training on them as-is presented two problems. First, some of these models reason in very long chains that sometimes fall into repetitive loops, a habit the team didn’t want Nemotron 3.5 to acquire. Second, the SFT data should teach safe behavior, not another model’s writing style. More broadly, data close to the policy’s own distribution is easier to learn and causes less forgetting [3], so even at the SFT stage the team aimed to keep the training data as on-policy as possible. To do this, they had Nemotron 3.5 Super rewrite the reasoning in each trajectory in its own words, which shortened long chains while leaving every action unchanged [3,4].

Figure 3 shows the effect. As-is, the open-model rollouts were nearly twice as surprising to the base model as its own rollouts (×1.99 perplexity). A policy fine-tuned on them drifted to ×1.40, while a policy fine-tuned on the rephrased rollouts stayed closer, at ×1.23. Rephrasing cut the added perplexity from 40% to 23%.

Figure 4: Rephrasing the SFT data halves the checkpoint’s reasoning length and nearly eliminates the most common repetitive fillers (e.g., “Hmm. OK”).

The SFT checkpoint trained on the rewritten data matched the quality of the one trained on the original rollouts, while generating far fewer tokens (Figure 4) and not regressing in other tasks.

Reward shaping in online RL

Once the team had established a good SFT trained Nemotron-3.5-Super VL checkpoint, with improved results and more diverse rollouts, they were able to continue to the second phase: online RL training using GRPO. 

Figure 5: Adding one reward change at a time from the same SFT checkpoint: removing STD normalization lifts reward above the repetition penalty alone, and adding the length penalty keeps reasoning from growing while reaching the highest reward

Vanilla GRPO computes each rollout’s advantage in two steps: it subtracts the group’s mean reward, then divides by the group’s standard deviation. We quickly found that the second step was a poor fit for Alice’s RL gyms, which use dense, continuous rewards. Dividing by the group’s spread throws away the size of the reward gap. For example, a group with rewards [0.49, 0.51] and a group with rewards [0, 1] both end up with advantages [−1, +1]. The ranking survives, but the magnitude is lost. With dense rewards, this means tiny and possibly noisy differences, such as a slight shift in an LLM judge’s score, get amplified into full-strength gradients [6,9]. We therefore dropped the standard-deviation normalization and kept only the group-mean baseline (Figure 5).

Another reward modification was needed to keep response length under control. Response length often grows during RL [5,6]. However, in this use case it grew sharply, and some rollouts looped until they used up the context window. To counter this, the team ablated two additional reward terms: (1) a group-relative length penalty, which penalizes the longest rollouts within each group [7,8], and (2) an n-gram repetition penalty, which penalizes rollouts whose reasoning repeats the same n-gram many times [5]. Indeed, the combination of these three modifications yielded the best results – reaching the highest quality, while generating the shortest response length. (Figure 5).

Results

Figure 6: In-distribution scores on the validation set and its GitHub and Slack subsets at each training stage, from the base model through SFT to GRPO.

The post-trained model was evaluated on two different sets. Using two lets us check whether its gains reflect real safety behavior rather than overfitting to the training environments. The first is a validation set, which uses the same GitHub and Slack environments as training and measures in-distribution performance. The second is CHASM, Alice’s Enterprise Indirect Prompt Injection (IPI) benchmark, designed as an out-of-distribution gym comprising 117 attack tasks across enterprise workflows spanning domains such as finance, HR, legal, and customer support, as well as industries from healthcare to telecom. 

CHASM shares no tasks or apps with the RL gyms, so it tests whether that behavior generalizes. Its attacks, mostly indirect prompt injection and poisoned records, try to push the agent into exfiltrating data, breaching confidentiality or diverting payments.

On the validation set, each stage builds on the last (Figure 6). SFT nearly doubles the score, from 35.19 to 65.28, and GRPO lifts it to 76.2. Most of the GRPO gain is on GitHub (66.47 → 85.17), while Slack improves only slightly (64.08 → 67.22).

The gains also carry over out of distribution. On CHASM, the post-trained model scores 33.96, up from 7.54 for the base model: about 4.5 times higher on environments and attack types it never saw in training.

Reaching frontier-level attack resilience with a custom post-training recipe

Together, AI21 and Alice improved Nemotron 3.5 Super VL’s attack resilience by 41 points above the base model and 10 points above GPT-6.1 Sol without harming overall task performance – and while costing 25x less than the frontier. AI21’s FDR team achieved these results by designing a custom training recipe that sequentially applied on-policy SFT, followed by a modified GRPO (online RL) technique.

This work demonstrates that companies can train open models that integrate their core safety and security policies as core behavioral principles, ensuring they complete their assigned tasks while spotting and resisting threats, such as prompt injection attacks.

References

  1. Harbor Framework Team. Harbor: A Framework for Evaluating and Optimizing Agents and Models in Container Environments. Zenodo, doi:10.5281/zenodo.20953922, 2026.
  2. DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948, 2025.
  3. Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, Qian Liu. Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning. ACL 2024, arXiv:2402.13669.
  4. Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman. STaR: Bootstrapping Reasoning With Reasoning. NeurIPS 2022, arXiv:2203.14465.
  5. Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, Xiang Yue. Demystifying Long Chain-of-Thought Reasoning in LLMs. arXiv:2502.03373, 2025.
  6. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin. Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783, 2025.
  7. Kimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599, 2025.
  8. Qiying Yu, Zheng Zhang, Ruofei Zhu, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025.
  9. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. arXiv:2402.14740, 2024.