He Co-Invented the Transformer. Now: Continuous Thought Machines - Llion Jones and Luke Darlow [Sakana AI]

The Transformer architecture (which powers ChatGPT and nearly all modern AI) might be trapping the industry in a localized rut, preventing us from finding true intelligent reasoning, according to the person who co-invented it. Llion Jones and Luke Darlow, key figures at the research lab Sakana AI, join the show to make this provocative argument, and also introduce new research which might lead the way forwards. **SPONSOR MESSAGES START** — Build your ideas with AI Studio from Google - — Tufa AI Labs is hiring ML Research Engineers — cyber•Fund is a founder-led investment firm accelerating the cybernetic economy Hiring a SF VC Principal: Submit investment deck: — **END** The "Spiral" Problem – Llion uses a striking visual analogy to explain what current AI is missing. If you ask a standard neural network to understand a spiral shape, it solves it by drawing tiny straight lines that just happen to look like a spiral. It "fakes" the shape without understanding the concept of spiraling. Introducing the Continuous Thought Machine (CTM) Luke Darlow deep dives into their solution: a biology-inspired model that fundamentally changes how AI processes information. The Maze Analogy: Luke explains that standard AI tries to solve a maze by staring at the whole image and guessing the entire path instantly. Their new machine "walks" through the maze step-by-step. Thinking Time: This allows the AI to "ponder." If a problem is hard, the model can naturally spend more time thinking about it before answering, effectively allowing it to correct its own mistakes and backtrack—something current Language Models struggle to do genuinely. TRANSCRIPT: TOC: 00:00:00 - Stepping Back from Transformers 00:00:43 - Introduction to Continuous Thought Machines (CTM) 00:01:09 - The Changing Atmosphere of AI Research 00:04:13 - Sakana’s Philosophy: Research Freedom 00:07:45 - The Local Minimum of Large Language Models 00:18:30 - Representation Problems: The Spiral Example 00:29:12 - Technical Deep Dive: CTM Architecture 00:36:00 - Adaptive Computation & Maze Solving 00:47:15 - Model Calibration & Uncertainty 01:00:43 - Sudoku Bench: Measuring True Reasoning REFS: Why Greatness Cannot be planned [Kenneth Stanley] The Hardware Lottery [Sara Hooker] Continuous Thought Machines [Luke Darlow et al / Sakana] LSTM: The Comeback Story? [Prof. Sepp Hochreiter] Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis [Kumar/Stanley] A Spline Theory of Deep Networks [Randall Balestriero] On the Biology of a Large Language Model [Anthropic, Jack Lindsey et al] The ARC Prize 2024 Winning Algorithm [Daniel Franzen and Jan Disselhoff] “The ARChitects” Neural Turing Machine [Graves] Adaptive Computation Time for Recurrent Neural Networks [Graves] Sudoko Bench [Sakana]

You can listen to He Co-Invented the Transformer. Now: Continuous Thought Machines - Llion Jones and Luke Darlow [Sakana AI] online on Radio and Podcast. Open the player on this page to stream the available audio.

He Co-Invented the Transformer. Now: Continuous Thought Machines - Llion Jones and Luke Darlow [Sakana AI] is an episode from Machine Learning Street Talk by Machine Learning Street Talk (MLST).

This episode is 01:12:39 long.

This episode was published on Nov 23, 2025.

Yes. Use the heart button on the episode page to add it to your favorite episodes list.

Yes. This page shows related episodes from Machine Learning Street Talk when more episodes are available from the podcast feed.