One token to corrupt them all: a vLLM debugging tale

TL;DR

While working on a new Jamba model, we noticed it would generate complete gibberish, but just once out of every thousand prompts. Finding and solving the bug sent us deep into the heart of vLLM, eventually touching on how its scheduler interacts with different model architectures. In addition to sharing our fix below, we also share the lessons we learned along the way about debugging a massive codebase like vLLM. We hope this detailed walkthrough makes your next vLLM debugging session a little less daunting.