Is It Time to Rethink LLM Pre-Training? with Aditi Raghunathan - #747

Today, we're joined by Aditi Raghunathan, assistant professor at Carnegie Mellon University, to discuss the limitations of LLMs and how we can build more adaptable and creative models. We dig into her ICML 2025 Outstanding Paper Award winner, “Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction,” which examines why LLMs struggle with generating truly novel ideas. We dig into the "Roll the dice" approach, which encourages structured exploration by injecting randomness at the start of generation, and the "Look before you leap" concept, which trains models to take "leaps of thought" using alternative objectives to create more diverse and structured outputs. We also discuss Aditi’s papers exploring the counterintuitive phenomenon of "catastrophic overtraining," where training models on more data improves benchmark performance but degrades their ability to be fine-tuned for new tasks, and dig into her lab's work on creating more controllable and reliable models, including the concept of "memorization sinks," an architectural approach to isolate and enable the targeted unlearning of specific information. The complete show notes for this episode can be found at

You can listen to Is It Time to Rethink LLM Pre-Training? with Aditi Raghunathan - #747 online on Radio and Podcast. Open the player on this page to stream the available audio.

Is It Time to Rethink LLM Pre-Training? with Aditi Raghunathan - #747 is an episode from The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) by TWIML.

This episode is 58:26 long.

This episode was published on Sep 16, 2025.

Yes. Use the heart button on the episode page to add it to your favorite episodes list.

Yes. This page shows related episodes from The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) when more episodes are available from the podcast feed.